FabuLight-ASD: Unveiling Speech Activity via Body Language

Hugo Carneiro; Stefan Wermter

REVIEW 3 major objections 4 minor 55 references

FabuLight-ASD: Unveiling Speech Activity via Body Language

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Body pose skeletons boost active speaker detection to 94.3% mAP.

desk verdict Solid incremental extension of Light-ASD with a pose stream; the 0.6 mAP gain is plausible but not statistically grounded, and the efficiency analysis is the strongest part. read the letter →

arxiv 2411.13674 v2 pith:4D6IUQGW submitted 2024-11-20 cs.CV cs.LGcs.NEcs.SDeess.AS

Hugo Carneiro , Stefan Wermter This is my paper · ORCID

classification cs.CVcs.LGcs.NEcs.SDeess.AS

keywords activespeakerdetectionbodyposeskeletongraphST-GCNmultimodalfusionlightweightmodelWASDdatasetmeanaverageprecision

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that body pose information, in the form of skeleton graphs, improves active speaker detection when added to a lightweight face-and-audio model. It builds on Light-ASD and adds a body stream whose feature encoder is built from spatial-temporal graph convolutions. On the WASD benchmark the extended model reaches 94.3% overall mAP, up from 93.7% for Light-ASD, with the largest gains in speech impairment, face occlusion, and human voice noise. The extra cost is modest: 27.3% more parameters and at most 2.4% more MAC operations. If true, this makes body language a cheap and useful cue for finding who is speaking in real-world scenes.

What carries the argument

The central mechanism is the body feature encoder: a three-block spatial-temporal graph convolutional network (ST-GCN) that treats a person's joints across frames as a graph, with within-frame skeletal edges and temporal edges connecting corresponding joints between adjacent frames. Each block uses two paths with spatial kernel sizes 3 and 5, defined by spatial configuration partitions around the nose as the central node, followed by temporal convolutions; the learned adjacency matrices are normalised and summed via tensor contraction. This encoder turns a $3 \times N_b \times T_f$ skeleton sequence into a $128 \times T_f$ feature tensor that aligns with the face and audio features, allowing fusion by element-wise sum.

What would settle it

Re-run the comparison on a version of WASD where no clip appears in both training and validation splits; if FabuLight-ASD's overall mAP drops to or below Light-ASD's, the claimed benefit of body pose is not established.

Watch

Extended reading notes

Core claim

The paper claims that a target person's body pose, encoded as a spatial-temporal skeleton graph, carries signal about whether that person is actively speaking, and that this signal can be fused with face and audio features within a lightweight architecture. FabuLight-ASD feeds COCO-format joints into a body feature encoder made of ST-GCN blocks, averages the joint dimension, and then sums the body feature tensor with the face and audio tensors before a BiGRU and a classification head. On WASD the upper-body variant scores 94.3% mAP overall, 0.6 points above Light-ASD, while the whole-body variant scores 94.0%; both beat all comparison models in overall mAP. The improvement is concentrated in categories where faces are impaired, occluded, or competing with human voice noise, which is exactly where pose cues should matter.

Load-bearing premise

The result assumes that the WASD train/validation protocol, which lets the same video clip appear in both splits for different target people, does not inflate the measured mAP of the target-only model.

Editorial extensions

If this is right

If the central claim holds, active speaker detection in social robots and embedded devices can gain robustness from body pose without a costly second image stream.
Pose information should help most in small-face, occluded-face, and noisy-audio conditions, which are exactly the cases where face-only and audio-only cues fail.
The upper-body variant reaching the highest overall mAP suggests that hand and arm motion, not leg motion, carries most of the speaking signal.
The small MAC increase means the pose stream could be added to existing Light-ASD deployments with minimal latency impact.
Because the body encoder aligns with the face/audio feature dimensions, the same fusion and temporal modeling machinery transfers to other multimodal tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported 0.6 mAP gain may be optimistic: WASD allows the same video clip in both training and validation splits with different target persons, and since the audio stream is identical for every person in a clip, part of the gain could come from memorising clip-specific audio rather than from pose.
A stronger test would be to re-split WASD so clips never cross splits, or to evaluate on AVA-ActiveSpeaker, where the model's target-only design avoids the context-leakage problem but pose annotations are absent.
Because pose-estimation confidence is a known failure mode, an adaptive weighting of the body stream by average joint confidence could push the gains higher, especially in surveillance settings.
The skeleton representation is compact enough that a real-time robot deployment could run pose estimation on a downscaled frame and still keep the detection pipeline interactive.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this is a clean, honest incremental extension of Light-ASD with an ST-GCN body-pose stream, and the efficiency story is the most solid part. The headline 0.6 mAP gain on WASD is real in the sense that their table shows it, but it is not statistically grounded and it concentrates in human-voice-noise and medium-face conditions. The paper's own breakdown shows the pose stream hurting in small-face and surveillance subsets. I would not call the central claim deceptive, but I would not call it proven either.

What is genuinely new: combining a lightweight face/audio ASD model with a skeleton-GCN body encoder is a natural and previously missing combination, and the paper does a thorough job of ablating where pose helps (HVN, face occlusion, medium resolution, 4+ speakers) and where it does not (surveillance, low-pose-confidence). The efficiency analysis is genuinely useful: +27.3% parameters and +2.4% MACs is a modest price, and they honestly note that HRNet preprocessing is excluded. The writing is clear and the code/weights are promised, which matters.

Soft spots, in rough order of importance:

1. No error bars or multi-seed runs anywhere. The overall improvement is 0.6 mAP; with single runs, that could easily be noise. Table 2 actually contains multiple cells where both pose variants underperform Light-ASD, so the aggregate gain is fragile and depends on a few categories. This is the main fix needed.

2. The WASD split overlap is acknowledged but dismissed too quickly. The shared-audio concern is legitimate: the audio stream is identical for every target in a clip, so a model could memorize clip-level audio characteristics during training and benefit on validation targets from the same clip. However, since Light-ASD also uses audio, this leakage would inflate both models roughly equally. It is more of a threat to the absolute numbers and to the comparison against Roxo's published results than to the incremental pose gain. Still, a per-clip split or leakage analysis would strengthen the paper.

3. Minor: the parameter table says '1,021 million' and '1,300 million' where the intended values are almost certainly 1.021M and 1.300M. That comma-decimal convention is confusing in an English-language paper. Also, Table 1's caption says all results were obtained by Roxo et al. even though the FabuLight-ASD rows are the authors' own, which is sloppy.

Who this is for: anyone working on lightweight ASD for social robotics or video conferencing. It is a solid, honest incremental contribution, not a paradigm shift. I would send it to peer review, with the expectation that the authors add multi-seed statistics and address the clip-overlap issue. As it stands, the efficiency claim is the most reliable part; the accuracy claim is plausible but under-supported.

Referee Report

3 major / 4 minor

Summary. The paper proposes FabuLight-ASD, an extension of the Light-ASD active speaker detection model that adds a skeleton-based body pose stream (upper-body or whole-body) encoded via spatial-temporal graph convolutions. The model is evaluated on the WASD dataset, reporting an overall mAP of 94.3% for the upper-body variant versus 93.7% for the Light-ASD baseline, with a modest increase in parameter count and MACs. The paper includes performance breakdowns by face resolution, pose-estimation confidence, number of speakers, and input temporal span, and releases code and model weights.

Significance. If the reported improvement is robust, the paper makes a useful incremental contribution: it demonstrates that a cheap skeleton-based pose stream can improve active speaker detection in some challenging conditions (notably human-voice noise, speech impairment, and face occlusion) without substantial computational overhead. The architecture description is clear, the code and weights are publicly available, and the authors are transparent about several dataset limitations and about subsets where pose information does not help. The main weakness is that the central performance claim rests on a small overall mAP gain that is not supported by error bars or significance testing, and the WASD train/validation overlap is dismissed too quickly for a model with a shared audio input.

major comments (3)

[Section 5.2, Table 1] The headline claim that FabuLight-ASD outperforms Light-ASD rests on a single evaluation, with no standard deviations, confidence intervals, or significance tests reported for the 0.6 mAP overall gain (94.3 vs 93.7). This is load-bearing because Table 2 shows multiple subsets where the pose variants are worse than Light-ASD, for example the small-face-resolution row (Light-ASD 81.7 vs upper-body 80.2 and whole-body 81.7) and the low whole-body-confidence row (Light-ASD 82.8 vs whole-body 80.3). Without uncertainty quantification, the central claim that body pose improves ASD is not statistically established.
[Section 2.1] The argument that the WASD train/validation overlap does not affect target-only models is incomplete because the audio input is the full scene audio and is identical for every target in a clip. If a validation target appeared as a non-target speaker in a training clip, or if the same clip appears in both splits, the model can learn clip-specific audio characteristics, including the voices of validation-target individuals, even though it ignores context individuals visually. This could inflate the reported 0.6 mAP gain. Please address this by evaluating on a subset with no clip-level overlap between training and validation, or by ablating the audio stream, and report whether the improvement persists.
[Table 1 vs Table 2] The Light-ASD numbers in Table 1 are taken from Roxo et al. [32] rather than obtained by the authors under the same training protocol used for FabuLight-ASD, while Table 2 appears to use a re-run Light-ASD baseline. This makes it unclear whether the overall comparison in Table 1 reflects an architectural benefit or differences in training details (for example, the different loss weights in Eq. 8 and the training schedule). Please clarify which entries are from the authors' own runs and, ideally, retrain Light-ASD under identical conditions when reporting the headline comparison.

minor comments (4)

[Section 5.4, Table 3] The text states that Light-ASD has 'a small number of parameters – 1,021 million,' which reads as 1.021 billion and is inconsistent with the claim of being lightweight. Please clarify whether the units in Table 3 are millions or thousands, and correct the corresponding sentences in Section 5.4.
[Table 1 caption] The caption says 'All results were obtained by Roxo et al. [32]' but the FabuLight-ASD rows were evidently evaluated by the authors; please specify that only the baseline rows come from Roxo et al.
[Equation 5] The graph distance d(v_{t,i}, v_{t,j}) used in Equation 5 is not formally defined; please state that it is the shortest-path length on the spatial skeleton graph within a frame.
[Throughout] There are several typographical inconsistencies, such as 'W ASD' with a space between W and ASD and 'T able 1' / 'T able 2' with a space after 'T'. Please unify the notation and proofread the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central mAP comparison is an empirical benchmark evaluation, not a derivation from its own inputs.

full rationale

The paper's central assertion, that FabuLight-ASD reaches 94.3% mAP and outperforms Light-ASD's 93.7% mAP on WASD, is a direct empirical benchmark result rather than a quantity derived from an assumed input. The body-pose stream is a newly added architectural component trained and evaluated on an external dataset, and the comparison numbers are either produced by the authors' own runs or attributed to the external evaluation by Roxo et al. [32]. The hand-set hyperparameters, such as the auxiliary loss weights 0.25 and 0.25 in Equation 8 and the temperature schedule in Equation 2, are training choices validated on the same validation split by standard practice; they are not renamed predictions and do not force the reported gains. The two self-citations ([5] and [6]) appear only in background and motivation contexts and are not load-bearing for the central claim. The paper itself acknowledges the WASD train/validation clip overlap and argues that target-only models are not adversely affected; whether that argument is accepted is a benchmark-validity or correctness concern, not a circularity of the derivation. Because no reduction of a claimed result to its own inputs or to a self-citation chain can be exhibited, the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on empirical training rather than derivation. The main free knobs are the auxiliary loss weights and the inherited temperature schedule. The key domain assumptions are the speech-body correlation, the accuracy of HRNet pose estimates, the absence of split leakage in WASD, and the statistical stability of the reported point estimates.

free parameters (2)

Auxiliary loss weights (face, body) = 0.25, 0.25
Eq. 8 sets L_total = L_fab + 0.25 L_f + 0.25 L_b. These weights are chosen by hand and no sensitivity analysis is provided; they affect the trained model and the reported mAP.
Temperature schedule constants = tau = 1.3 - 0.02 * epoch
Eq. 2, inherited from Light-ASD. It influences training dynamics but is not tuned here.

assumptions (4)

domain assumption Body pose correlates with speech activity in natural conversation.
Sections 1 and 2 motivate the pose stream by citing GestSync [16] and Vajaria et al. [39]. If this correlation is weak in WASD, the pose stream cannot help.
domain assumption HRNet (MMPose) provides sufficiently accurate joint coordinates from WASD body bounding boxes.
Section 4.1 uses HRNet to estimate COCO joints. Pose estimation errors could add noise or bias; the paper analyzes confidence levels but does not test alternative pose estimators.
domain assumption The WASD train/val split overlap does not inflate performance for target-only models.
Section 2.1 asserts the overlap is harmless because the model ignores context individuals, but the audio stream is shared across targets in the same clip, creating a potential leakage path.
domain assumption The single-run mAP difference is not due to random seed variation.
The paper reports point estimates from one training run per model with no repeated seeds or significance testing, so statistical noise is not ruled out.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FabuLight-ASD: Unveiling Speech Activity via Body Language." pith.science (2026). https://pith.science/paper/4D6IUQGW

@misc{pith2026241113674,
  author       = {Pith},
  title        = {Pith review of: FabuLight-ASD: Unveiling Speech Activity via Body Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4D6IUQGW}},
  note         = {Machine review of arXiv:2411.13674}
}

read the original abstract

Active speaker detection (ASD) in multimodal environments is crucial for various applications, from video conferencing to human-robot interaction. This paper introduces FabuLight-ASD, an advanced ASD model that integrates facial, audio, and body pose information to enhance detection accuracy and robustness. Our model builds upon the existing Light-ASD framework by incorporating human pose data, represented through skeleton graphs, which minimises computational overhead. Using the Wilder Active Speaker Detection (WASD) dataset, renowned for reliable face and body bounding box annotations, we demonstrate FabuLight-ASD's effectiveness in real-world scenarios. Achieving an overall mean average precision (mAP) of 94.3%, FabuLight-ASD outperforms Light-ASD, which has an overall mAP of 93.7% across various challenging scenarios. The incorporation of body pose information shows a particularly advantageous impact, with notable improvements in mAP observed in scenarios with speech impairment, face occlusion, and human voice background noise. Furthermore, efficiency analysis indicates only a modest increase in parameter count (27.3%) and multiply-accumulate operations (up to 2.4%), underscoring the model's efficiency and feasibility. These findings validate the efficacy of FabuLight-ASD in enhancing ASD performance through the integration of body pose data. FabuLight-ASD's code and model weights are available at https://github.com/knowledgetechnologyuhh/FabuLight-ASD.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 34 canonical work pages

[32]

IEEE Transactions on Biometrics, Behavior, and Identity Science https://doi

Roxo T, Costa JC, In´ acio PRM, et al (2024) W ASD: A wilder active speaker detection dataset. IEEE Transactions on Biometrics, Behavior, and Identity Science https://doi. org/10.1109/TBIOM.2024.3412821

arXiv 2024
[1]

In: Yegnanarayana B (ed) Interspeech 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India, 2-6 Septem- ber 2018

Afouras T, Chung JS, Zisserman A (2018) The conversation: Deep audio-visual speech enhancement. In: Yegnanarayana B (ed) Interspeech 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India, 2-6 Septem- ber 2018. ISCA, pp 3244–3248, https://doi. org/10.21437/INTERSPEECH.2018-1400

work page doi:10.21437/interspeech.2018-1400 2018
[2]

In: 2020 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2020, Seattle, W A, USA, June 13-19, 2020

Alc´ azar JL, Caba F, Mai L, et al (2020) Active speakers in context. In: 2020 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2020, Seattle, W A, USA, June 13-19, 2020. Computer Vision Founda- tion / IEEE, pp 12462–12471, https: //doi.org/10.1109/CVPR42600.2020.01248

arXiv 2020
[4]

Alc´ azar JL, Cordes M, Zhao C, et al (2022) End-to-end active speaker detection. In: Avi- dan S, Brostow GJ, Ciss´ e M, et al (eds) 19 Computer Vision - ECCV 2022 - 17th Euro- pean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, Lec- ture Notes in Computer Science, vol 13697. Springer, pp 126–143, https://doi.org/10. 1007/978...

work page 2022
[5]

Carneiro HCC, Weber C, Wermter S (2021) FaVoA: Face-voice association favours ambiguous speaker detection. In: Farkaˇ s I, Masulli P, Otte S, et al (eds) Artificial Neu- ral Networks and Machine Learning - ICANN 2021 - 30th International Conference on Arti- ficial Neural Networks, Bratislava, Slovakia, September 14-17, 2021, Proceedings, Part I, Lecture N...

work page doi:10.1007/978-3-030-86362-3 2021
[6]

Neu- rocomputing 545:126271

Carneiro HCC, Weber C, Wermter S (2023) Whose emotion matters? Speaking activity localisation without prior knowledge. Neu- rocomputing 545:126271. https://doi.org/10. 1016/J.NEUCOM.2023.126271

work page arXiv 2023
[7]

In: Zhang Z, Cohen P, Bohus D, et al (eds) Pro- ceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seat- tle, W A, USA, November 09 - 13, 2015

Chakravarty P, Mirzaei S, Tuytelaars T, et al (2015) Who’s speaking?: Audio-supervised classification of active speakers in video. In: Zhang Z, Cohen P, Bohus D, et al (eds) Pro- ceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seat- tle, W A, USA, November 09 - 13, 2015. ACM, pp 87–90, https://doi.org/10.1145/2818346. 2820780

work page doi:10.1145/2818346 2015
[8]

Cho K, van Merrienboer B, G¨ ul¸ cehre C ¸ , et al (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Moschitti A, Pang B, Daelemans W (eds) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meet- ing of SIGDAT, a Sp...

doi:10.3115/v1/d14-1179 2014

Show all 55 references

[9]

URL https://static.googleusercontent.com/ media/research.google.com/en//ava/2019/ Naver Corporation.pdf

Chung JS (2019) Naver at Activ- ityNet challenge 2019 - Task B active speaker detection (A V A). URL https://static.googleusercontent.com/ media/research.google.com/en//ava/2019/ Naver Corporation.pdf

2019
[10]

In: Kubin G, Kacic Z (eds) Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019

Chung JS, Lee B, Han I (2019) Who said that?: Audio-visual speaker diarisation of real-world meetings. In: Kubin G, Kacic Z (eds) Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019. ISCA, pp 371–3...

2019 doi
[11]

In: Meng H, Xu B, Zheng TF (eds) Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020

Chung JS, Huh J, Nagrani A, et al (2020) Spot the conversation: Speaker diari- sation in the wild. In: Meng H, Xu B, Zheng TF (eds) Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 202...

2020
[12]

In: 2000 IEEE Interna- tional Conference on Multimedia and Expo, ICME 2000, New York, NY, USA, July 30 - August 2, 2000

Cutler R, Davis LS (2000) Look who’s talking: Speaker detection using video and audio correlation. In: 2000 IEEE Interna- tional Conference on Multimedia and Expo, ICME 2000, New York, NY, USA, July 30 - August 2, 2000. IEEE Computer Soci- ety, pp 1589–1592, https://doi.org/10...

2000
[13]

In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022

Datta G, Etchart T, Yadav V, et al (2022) ASD-Transformer: Efficient active speaker detection using self and multimodal trans- formers. In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022. IEEE, pp 45...

2022
[14]

https://doi.org/10

Donley J, Tourbabin V, Lee J, et al (2021) EasyCom: An augmented reality dataset to support algorithms for easy communication in noisy environments. https://doi.org/10. 48550/ARXIV.2107.04174

2021 doi arXiv
[15]

In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, 20 USA, June 18-24, 2022

Grauman K, Westbury A, Byrne E, et al (2022) Ego4D: Around the world in 3,000 hours of egocentric video. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, 20 USA, June 18-24, 2022. IEEE, pp 18973– 18990, https://doi.org/10.1109/CVP...

2022
[16]

In: 34th British Machine Vision Conference 2023, BMVC 2023, Aberdeen, UK, November 20-24, 2023

Hegde SB, Zisserman A (2023) GestSync: Determining who is speaking without a talk- ing head. In: 34th British Machine Vision Conference 2023, BMVC 2023, Aberdeen, UK, November 20-24, 2023. BMV A Press, pp 506–509, URL http://proceedings.bmvc2023. org/506/

2023
[17]

https://doi.org/10.48550/ARXIV.1704

Howard AG, Zhu M, Chen B, et al (2017) MobileNets: Efficient convolutional neural networks for mobile vision applications. https://doi.org/10.48550/ARXIV.1704. 04861

2017 doi
[18]

Ioffe S, Szegedy C (2015) Batch normal- ization: Accelerating deep network train- ing by reducing internal covariate shift. In: Bach FR, Blei DM (eds) Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, JMLR Workshop ...

2015
[19]

Jiang Y, Tao R, Pan Z, et al (2023) Tar- get active speaker detection with audio-visual cue. In: Harte N, Carson-Berndsen J, Jones G (eds) Interspeech 2023, 24th Annual Con- ference of the International Speech Commu- nication Association, Dublin, Ireland, 20 - 24 August 2023. ...

2023 doi
[20]

In: ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 8391–8395, https://doi.org/10

Jung C, Lee S, Nam K, et al (2024) TalkNCE: Improving active speaker detection with talk- aware contrastive learning. In: ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 8391–8395, https://doi.org/10. 1109/ICASSP48485.20...

2024
[21]

Kim YJ, Heo H, Choe S, et al (2021) Look who’s talking: Active speaker detec- tion in the wild. In: Hermansky H, Cer- nock´ y H, Burget L, et al (eds) Interspeech 2021, 22nd Annual Conference of the Inter- national Speech Communication Association, Brno, Czechia, 30 August - 3...

2021
[22]

In: 2021 IEEE/CVF Interna- tional Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17,

K¨ op¨ ukl¨ u O, Taseska M, Rigoll G (2021) How to design a three-stage architecture for audio-visual active speaker detection in the wild. In: 2021 IEEE/CVF Interna- tional Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17,

2021
[23]

In: IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24,

Liao J, Duan H, Feng K, et al (2023) A light weight model for active speaker detec- tion. In: IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24,

2023
[24]

IEEE, pp 1173–1183, https://doi.org/ 10.1109/ICCV48922.2021.00123

2021
[25]

Min K, Roy S, Tripathi S, et al (2022) Learning long-term spatial-temporal graphs for active speaker detection. In: Avidan S, Brostow GJ, Ciss´ e M, et al (eds) Com- puter Vision - ECCV 2022 - 17th Euro- pean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part...

2022
[26]

URL https://github.com/open-mmlab/ mmpose

MMPose Contributors (2020) OpenMMLab pose estimation toolbox and benchmark. URL https://github.com/open-mmlab/ mmpose

2020
[27]

Lin T, Maire M, Belongie SJ, et al (2014) Microsoft COCO: Common objects in con- text. In: Fleet DJ, Pajdla T, Schiele B, et al (eds) Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part V, Lecture Notes in Comp...

2014 doi
[28]

IEEE Transactions on Multimedia 24:942–954

Qian X, Brutti A, Lanz O, et al (2022) Audio- visual tracking of concurrent speakers. IEEE Transactions on Multimedia 24:942–954. https://doi.org/10.1109/TMM.2021.3061800

2022
[29]

In: Meng H, Xu B, Zheng TF (eds) Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Vir- tual Event, Shanghai, China, 25-29 October

Qu L, Weber C, Wermter S (2020) Multi- modal target speech separation with voice and face references. In: Meng H, Xu B, Zheng TF (eds) Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Vir- tual Event, Shanghai, China, 25-29 October

2020
[30]

In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, 21 Toronto, ON, Canada, June 6-11, 2021

Qian X, Madhavi MC, Pan Z, et al (2021) Multi-target DoA estimation with an audio-visual fusion mechanism. In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, 21 Toronto, ON, Canada, June 6-11, 2021. IEEE, pp 4280–4284, https://doi.org/10...

2021
[31]

In: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8,

Roth J, Chaudhuri S, Klejch O, et al (2020) A V A Active Speaker: An audio- visual dataset for active speaker detection. In: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8,

2020
[33]

In: IEEE Winter Con- ference on Applications of Computer Vision, W ACV 2021, Waikoloa, HI, USA, January 3-8, 2021

Shahid M, Beyan C, Murino V (2021) S- VV AD: Visual voice activity detection by motion segmentation. In: IEEE Winter Con- ference on Applications of Computer Vision, W ACV 2021, Waikoloa, HI, USA, January 3-8, 2021. IEEE, pp 2331–2340, https://doi. org/10.1109/W ACV48630.2021.00238

2021
[34]

Multi- media Tools and Applications https: //doi.org/10.1007/s11042-024-18457-9

Radman A, Laaksonen J (2024) AS- Net: Active speaker detection using deep audio-visual attention. Multi- media Tools and Applications https: //doi.org/10.1007/s11042-024-18457-9

2024 doi
[35]

Stefanov K, Sugimoto A, Beskow J (2016) Look who’s talking: visual identification of the active speaker in multi-party human- robot interaction. In: Truong KP, Heylen D, Nishida T, et al (eds) Proceedings of the 2nd Workshop on Advancements in Social Signal Processing for Mult...

2016
[36]

IEEE, pp 4492–4496, https://doi.org/ 10.1109/ICASSP40776.2020.9053900

2020
[37]

In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019

Sun K, Xiao B, Liu D, et al (2019) Deep high- resolution representation learning for human pose estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, pp 5693–5703, https:...

2019
[38]

In: Shen HT, Zhuang Y, Smith JR, et al (eds) MM ’21: ACM Multimedia Confer- ence, Virtual Event, China, October 20 - 24,

Tao R, Pan Z, Das RK, et al (2021) Is some- one speaking?: Exploring long-term temporal features for audio-visual active speaker detec- tion. In: Shen HT, Zhuang Y, Smith JR, et al (eds) MM ’21: ACM Multimedia Confer- ence, Virtual Event, China, October 20 - 24,

2021
[39]

IEEE Transactions on Multime- dia 25:7825–7836

Sharma R, Somandepalli K, Narayanan S (2023) Cross modal video representations for weakly supervised active speaker local- ization. IEEE Transactions on Multime- dia 25:7825–7836. https://doi.org/10.1109/ TMM.2022.3229975

2023
[40]

In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seat- tle, W A, USA, June 17-21, 2024

Wang X, Cheng F, Bertasius G, et al (2024) LoCoNet: Long-short context net- work for active speaker detection. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seat- tle, W A, USA, June 17-21, 2024. Computer Vision Foundation / IEEE, pp 18462...

2024
[41]

In: Salvi G, Dupont S (eds) Pro- ceedings of GLU 2017 International Work- shop on Grounding Language Understanding

Stefanov K, Beskow J, Salvi G (2017) Vision- based active speaker detection in multiparty interaction. In: Salvi G, Dupont S (eds) Pro- ceedings of GLU 2017 International Work- shop on Grounding Language Understanding. ISCA, pp 47–51, https://doi.org/10.21437/ GLU.2017-10

2017
[42]

IEEE Transactions on Mul- timedia 25:5800–5812

Xiong J, Zhou Y, Zhang P, et al (2023) Look&listen: Multi-modal correlation learn- ing for active speaker detection and speech enhancement. IEEE Transactions on Mul- timedia 25:5800–5812. https://doi.org/10. 1109/TMM.2022.3199109

2023
[43]

Yan S, Xiong Y, Lin D (2018) Spatial tempo- ral graph convolutional networks for skeleton- based action recognition. In: McIlraith SA, Weinberger KQ (eds) Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications o...

2018
[44]

ACM, pp 3927–3935, https://doi.org/ 10.1145/3474085.3475587
[45]

IEEE Transactions on Circuits and Systems for Video Technology 18(11):1608–

Vajaria H, Sarkar S, Kasturi R (2008) Explor- ing co-occurence between speech and body movement for audio-guided video localiza- tion. IEEE Transactions on Circuits and Systems for Video Technology 18(11):1608–

2008
[46]

In: Shen HT, Zhuang Y, Smith JR, et al (eds) MM ’21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021

Zhang Y, Liang S, Yang S, et al (2021) UniCon: Unified context network for robust active speaker detection. In: Shen HT, Zhuang Y, Smith JR, et al (eds) MM ’21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021. ACM, pp 3964–3972, https://doi.org/10. 1145/...

2021
[47]

URL https://static.googleusercontent.com/ media/research.google.com/en//ava/2022/ S1 ICTCAS UCAS UniCon+.pdf 23

Zhang Y, Liang S, Yang S, et al (2022) UniCon+: ICTCAS-UCAS sub- mission to the A V A-ActiveSpeaker task at ActivityNet Challenge 2022. URL https://static.googleusercontent.com/ media/research.google.com/en//ava/2022/ S1 ICTCAS UCAS UniCon+.pdf 23

2022
[48]

In: 32nd IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2022, Xi’an, China, August 22-25,

Wuerkaixi A, Zhang Y, Duan Z, et al (2022) Rethinking audio-visual synchro- nization for active speaker detection. In: 32nd IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2022, Xi’an, China, August 22-25,

2022
[52]

URL https://static.googleusercontent.com/ media/research.google.com/en//ava/2019/ Multi Task Learning for Audio Visual Active Speaker Detection.pdf

Zhang Y, Xiao J, Yang S, et al (2019) Multi-task learning for audio- visual active speaker detection. URL https://static.googleusercontent.com/ media/research.google.com/en//ava/2019/ Multi Task Learning for Audio Visual Active Speaker Detection.pdf

2019
[53]

Zhang Y, Liang S, Yang S, et al (2021) ICTCAS-UCAS-TAL submission to the A V A- ActiveSpeaker task at ActivityNet Challenge

2021
[54]

com/media/research.google.com/en//ava/ 2021/S1 ICTCAS-UCAS-TAL.pdf

URL https://static.googleusercontent. com/media/research.google.com/en//ava/ 2021/S1 ICTCAS-UCAS-TAL.pdf

2021
[1617]

2005602 22

https://doi.org/10.1109/TCSVT.2008. 2005602 22

2008 doi
[2020]

ISCA, pp 1416–1420, https://doi.org/ 10.21437/INTERSPEECH.2020-1697

2020 doi
[2021]

ISCA, pp 3675–3679, https://doi.org/ 10.21437/INTERSPEECH.2021-2041

2021 doi
[2022]

IEEE, pp 1–6, https://doi.org/10.1109/ MLSP55214.2022.9943352

2022
[2023]

org/10.1109/CVPR52729.2023.02196

IEEE, pp 22932–22941, https://doi. org/10.1109/CVPR52729.2023.02196

2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [32]

IEEE Transactions on Biometrics, Behavior, and Identity Science https://doi

Roxo T, Costa JC, In´ acio PRM, et al (2024) W ASD: A wilder active speaker detection dataset. IEEE Transactions on Biometrics, Behavior, and Identity Science https://doi. org/10.1109/TBIOM.2024.3412821

arXiv 2024

[2] [1]

In: Yegnanarayana B (ed) Interspeech 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India, 2-6 Septem- ber 2018

Afouras T, Chung JS, Zisserman A (2018) The conversation: Deep audio-visual speech enhancement. In: Yegnanarayana B (ed) Interspeech 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India, 2-6 Septem- ber 2018. ISCA, pp 3244–3248, https://doi. org/10.21437/INTERSPEECH.2018-1400

work page doi:10.21437/interspeech.2018-1400 2018

[3] [2]

In: 2020 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2020, Seattle, W A, USA, June 13-19, 2020

Alc´ azar JL, Caba F, Mai L, et al (2020) Active speakers in context. In: 2020 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2020, Seattle, W A, USA, June 13-19, 2020. Computer Vision Founda- tion / IEEE, pp 12462–12471, https: //doi.org/10.1109/CVPR42600.2020.01248

arXiv 2020

[4] [4]

Alc´ azar JL, Cordes M, Zhao C, et al (2022) End-to-end active speaker detection. In: Avi- dan S, Brostow GJ, Ciss´ e M, et al (eds) 19 Computer Vision - ECCV 2022 - 17th Euro- pean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, Lec- ture Notes in Computer Science, vol 13697. Springer, pp 126–143, https://doi.org/10. 1007/978...

work page 2022

[5] [5]

Carneiro HCC, Weber C, Wermter S (2021) FaVoA: Face-voice association favours ambiguous speaker detection. In: Farkaˇ s I, Masulli P, Otte S, et al (eds) Artificial Neu- ral Networks and Machine Learning - ICANN 2021 - 30th International Conference on Arti- ficial Neural Networks, Bratislava, Slovakia, September 14-17, 2021, Proceedings, Part I, Lecture N...

work page doi:10.1007/978-3-030-86362-3 2021

[6] [6]

Neu- rocomputing 545:126271

Carneiro HCC, Weber C, Wermter S (2023) Whose emotion matters? Speaking activity localisation without prior knowledge. Neu- rocomputing 545:126271. https://doi.org/10. 1016/J.NEUCOM.2023.126271

work page arXiv 2023

[7] [7]

In: Zhang Z, Cohen P, Bohus D, et al (eds) Pro- ceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seat- tle, W A, USA, November 09 - 13, 2015

Chakravarty P, Mirzaei S, Tuytelaars T, et al (2015) Who’s speaking?: Audio-supervised classification of active speakers in video. In: Zhang Z, Cohen P, Bohus D, et al (eds) Pro- ceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seat- tle, W A, USA, November 09 - 13, 2015. ACM, pp 87–90, https://doi.org/10.1145/2818346. 2820780

work page doi:10.1145/2818346 2015

[8] [8]

Cho K, van Merrienboer B, G¨ ul¸ cehre C ¸ , et al (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Moschitti A, Pang B, Daelemans W (eds) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meet- ing of SIGDAT, a Sp...

doi:10.3115/v1/d14-1179 2014