Face-Voice Association for Audiovisual Active Speaker Detection in Egocentric Recordings

Jason Clarke; Stefan Goetze; Yoshihiko Gotoh

REVIEW 3 major objections 4 minor 36 references

Face-Voice Association for Audiovisual Active Speaker Detection in Egocentric Recordings

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Active speaker detection can be built purely from face-voice identity ties, matching heavier synchronisation-based systems in egocentric video while using far fewer learnable parameters.

desk verdict Face-voice association alone is a fresh angle for egocentric ASD, but the 'exceeding' claim rests on a filtered subset and oracle boundaries; on the standard protocol it trails LoCoNet and only ties SPELL. read the letter →

arxiv 2506.18055 v1 pith:VUT4VG2L submitted 2025-06-22 cs.MM cs.SDeess.AS

Jason Clarke , Yoshihiko Gotoh , Stefan Goetze This is my paper · ORCID

classification cs.MMcs.SDeess.AS

keywords activespeakerdetectionegocentricvideoface-voiceassociationaudiovisualdiarizationutterancesegmentationtransformerbiometric

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that active speaker detection—deciding which visible person is talking at a given moment—does not have to model the temporal synchronisation between lip motion and speech. Instead, it can be done by matching each speech segment to a face through learned face-voice identity associations. The proposed system, SL-ASD, pairs a face-voice association model with a transformer that pools face frames by quality, and a diarization front-end that carves the audio into single-speaker utterances. On the Ego4D egocentric benchmark it reaches 59.7 mAP with 0.4 million learnable parameters, compared with 51.0 to 68.4 mAP for synchronisation-based models that use 16 to 30 million parameters. The claim matters because egocentric recordings—head-mounted cameras, motion, occlusions, noise—degrade the fine visual and acoustic cues that synchronisation models rely on, while identity-based matching only needs one clearly visible frame of a face.

What carries the argument

The central mechanism is the substitution of temporal synchronisation modelling with a face-voice association space. The enabling component is the transformer-based frame-quality weighting: a single self-attention layer attends along the face-track sequence, learns to suppress blurred or occluded frames and emphasises the few clear frames that carry identity, then mean-pooling produces one quality-informed face embedding per identity. Cross-attention between this face embedding (query) and the utterance's audio embedding (keys/values) yields the speaker-activity probability. Because only one good frame is enough to establish identity, the system avoids the need for a majority of synchronised, crisp visual speech cues.

What would settle it

An ablation that replaces the transformer frame-quality weighting with simple mean pooling across face frames should measurably reduce mAP on Ego4D; if it does not, the dynamic quality-weighting mechanism is not load-bearing and the performance must come from the face-voice embedding alone.

Watch

Extended reading notes

Core claim

In egocentric recordings, where occlusion, motion blur and acoustic noise break the temporal alignment of audio and visual speech cues, the paper shows that an active speaker detector can be built entirely on cross-modal biometric associations rather than on audio-visual synchronisation. The system, SL-ASD, first segments the audio into speaker-invariant utterances using a diarization front-end, then embeds each utterance and each visible face track into a shared face-voice embedding space using a finetuned, frozen Self-Lifting model. A transformer encoder with learned self-attention weights each face frame by its visual quality before mean-pooling into one identity embedding per person, and cross-attention scores the match between each utterance and each visible identity. The resulting probability is assigned to the face tracks concurrent with the utterance. With only 0.4M learnable parameters, the system attains 59.7 mAP on the Ego4D validation fold—comparable to SPELL (60.7 mAP, 23M parameters) and below LoCoNet (68.4 mAP, 30M parameters)—and 80.1 mAP when utterance boundaries are given, beating all synchronisation baselines.

Load-bearing premise

The whole pipeline assumes the audio segmenter delivers clean, single-speaker utterances with high recall; any missed, spurious, or overlapping utterance corrupts the audio embedding and propagates uncorrected through the face-voice matching.

Editorial extensions

If this is right

If the claim holds, egocentric ASD can be implemented at a fraction of the parameter budget of synchronisation models, with competitive accuracy; at 0.4M learnable parameters the system is well within the range of on-device inference.
The ceiling of the approach is set by the segmentation front-end: with groundtruth utterance boundaries the system reaches 80.1 mAP, so improvements in speaker diarization transfer directly to ASD accuracy.
The dynamic frame-quality weighting is a direct reason the method tolerates degraded egocentric video: ablating it to simple frame-averaging would test its necessity, and the current results imply that at least one clear identity-bearing frame per utterance is sufficient.
In the dynamic subsets defined by front-end recall, SL-ASD outperforms TalkNet and Light-ASD regardless of which segmenter supplies the utterances, suggesting the identity-based matching itself is the source of the advantage, not merely a strong front-end.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The symmetry between face and voice in the embedding space suggests the same construction could be run in reverse: replace the utterance segmentation front-end with face-track clustering and query audio segments against visual identities, which would relax the requirement that utterance segmentation be speaker-pure.
Because the method relies on identity rather than synchrony, it should be nearly invariant to artificially desynchronised audio-video tracks; a direct test would be to misalign the audio by 0.5–1 s and measure whether SL-ASD's mAP changes, where synchronisation baselines are expected to collapse.
The 80.1 mAP with groundtruth boundaries indicates that the face-voice association step, not the segmentation, is the current bottleneck in the fully automatic pipeline; improving diarization recall for egocentric audio should raise end-to-end mAP considerably.
A testable extension would be to apply the same biometric-only substitution to exocentric benchmarks such as AVA-ActiveSpeaker; the paper does not claim parity there, and the expectation from its egocentric motivation is that synchronisation models would retain an advantage when visual speech cues are reliably visible.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the core idea is real and worth a second look, but the headline claim is not yet supported by the evidence. The new thing here is using face-voice identity matching alone, with a frozen Self-Lifting encoder plus a quality-weighted transformer aggregator, as the sole decision signal for egocentric ASD. Prior work only used such associations as a side-signal or for utterance attribution; this paper makes it the whole system, and that is a legitimate novelty. The engineering is clean, the writing is honest about the modular assumptions, and the code is public.

Where the paper gets into trouble is the comparison. The 'exceeding' results in Table I come from Ego4D†, a dynamic subset that only scores face tracks concurrent with detected utterances. That filters out exactly the failures that the paper itself says propagate through the pipeline. And the strongest number, 80.1 mAP, uses ground-truth utterance boundaries, which are not available in deployment. On the standard validation fold (Table II), SL-ASD with Pyannote reaches 59.7 mAP, below SPELL (60.7) and well below LoCoNet (68.4). So the claim that it 'exceeds' synchronization methods is contingent on a modified protocol and oracle information. This is not hidden—the paper states the protocol clearly—but the abstract and conclusion use the stronger claim.

The parameter accounting is also generous. The reported 0.4M learnable parameters exclude the frozen face encoder and the Pyannote segmenter, both of which are required for the system to work. The paper justifies this by modular-pipeline assumptions, and that is a defensible position, but it makes the '0.4M vs 30M' headline misleading.

The front-end segmenter is load-bearing. The paper explicitly acknowledges in Section II-B that segmentation errors propagate uncorrected. That is not a hidden flaw, but it means the system's practical performance is bounded by Pyannote's recall. The comparison to end-to-end sync models is therefore not apples-to-apples.

What the paper does well: it is transparent about its choices, it identifies the failure mode of the segmenter, and it offers a genuinely different design point for egocentric ASD. The results are plausible, and the approach deserves to be tested against a standard protocol. I would want error bars or at least multiple runs, and a version that reports the full-system parameter count.

Who this is for: researchers in egocentric ASD and audio-visual association. It would make a solid paper after revision. My recommendation: send it to peer review. The idea is novel enough and the reporting honest enough that referee time is warranted; the right reviewers can push for standard-protocol evaluation and a fairer parameter comparison.

Referee Report

3 major / 4 minor

Summary. The paper proposes SL-ASD, an audiovisual active speaker detection system for egocentric video that replaces conventional audiovisual synchronization modeling with face-voice identity association. The system uses a frozen Self-Lifting face-voice embedding model, a transformer encoder that aggregates face-track frames by visual quality, and an utterance segmentation front-end (Pyannote or Silero) to attribute detected utterances to visible identities. Experiments on the Ego4D validation fold report that SL-ASD achieves 59.7 mAP with the Pyannote front-end, which is comparable to SPELL (60.7) and below LoCoNet (68.4), while using only 0.4M learnable parameters. On dynamic subsets defined by the front-end recall, the paper claims performance exceeding synchronization baselines, with an oracle-boundary variant reaching 80.1 mAP.

Significance. The core idea of replacing synchronization-based ASD with biometric face-voice association in egocentric settings is interesting and, if properly validated, would be a useful alternative to current approaches. The paper provides code and reports a much smaller learnable parameter count than competing end-to-end models. However, the headline performance claims are currently entangled with a modified evaluation protocol and with the exclusion of integral pretrained components from the parameter and FLOP counts, so the significance as stated is not yet fully established.

major comments (3)

[Section IV, Table I; Section III-C] The 'exceeding' claim is based on the dynamic Ego4D† subsets, which discard all face tracks without concurrent detected utterances. This filtering removes exactly the failure cases the paper itself identifies in Section II-B as propagating uncorrected through the pipeline. Under the standard full-validation protocol in Table II, SL-ASD:Pyannote reaches 59.7 mAP, which is below LoCoNet (68.4) and slightly below SPELL (60.7). Please present the comparison on the full validation fold for all methods under a single scoring protocol, and adjust the claim that the approach exceeds synchronization-based methods.
[Section IV, Table I] The 'SL-ASD:Groundtruth' row (80.1 mAP) uses oracle utterance boundaries that are not available at deployment. This configuration is the only one that clearly exceeds all listed baselines, so the abstract's wording 'in certain cases exceeding' is misleading when the only such case relies on oracle segments. Please relegate this result to an upper-bound analysis and base the main comparison on the Pyannote or Silero front-end configurations.
[Section IV, Table II and parameter discussion] The parameter count of 0.4M and av-FLOPs of 0.21 GFLOPs exclude the frozen Inception-V1 face encoder and the Pyannote segmenter, which are integral to the system's operation. Comparing this partial count with the full parameter counts of end-to-end models (e.g., 30M for LoCoNet) overstates the efficiency advantage. Please report the total number of parameters and total inference FLOPs including all components, and rephrase the efficiency claim to state explicitly which components are excluded.

minor comments (4)

[Figure 1 caption] There is a typo: 'utteraces' should be 'utterances'. The caption would also benefit from a clearer explanation of the dotted-line notation for utterances.
[Section III-B and references] Reference [31] spells 'VoxCeleb' as 'V oxCeleb'; please correct. Also ensure consistent spelling of 'Inception-V1' throughout.
[Tables I and II] The TalkNet and Light-ASD scores in Table I differ across rows and from the values in Table II. Please clarify whether these baselines were re-evaluated on the dynamic subsets under the same protocol, or whether the numbers are from the original manuscripts; currently the table is difficult to interpret.
[Section IV, implementation details] 'its important to note' should be 'it is important to note'. The av-FLOPs values for SL-ASD should be clearly labeled as partial, given the exclusions described in the text.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the central mAP results are produced by a model trained on one fold and evaluated on a held-out fold; the dynamic Ego4D† subset and excluded parameters are explicit protocol caveats, not inputs that force the claimed outcome.

full rationale

The paper makes no first-principles derivation: it reports an empirical system (SL-ASD) trained on a stratified training subset of Ego4D-AVD and evaluated on the held-out validation fold. The claimed 'comparable/exceeding' performance is therefore not equivalent by construction to any training target. The adapted face-voice association model (Self-Lifting [19]) is an external, separately trained model; the transformer encoder and cross-attention are described with enough specificity to be tested independently. The two self-citations ([13], [15]) occur in the literature review and as provenance for the transformer aggregation idea; neither is used as a uniqueness theorem or to forbid alternative architectures, and the central evaluation does not depend on accepting those citations. The main caveats are evaluation-protocol choices, not circularity: Table I's Ego4D† is a dynamic subset in which only face tracks concurrent with detected utterances are scored, so SL-ASD's own missed detections, which the paper acknowledges propagate errors, are excluded from that particular table; and the 0.4M parameter count excludes the frozen face encoder and Pyannote segmenter, an exclusion the paper explicitly justifies. These affect the fairness of the comparison, but they do not make any reported number equal to an input or any prediction equivalent to a fitted parameter. No pattern from the enumerated circularity kinds is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central result rests on typical ML-system assumptions: clean single-speaker segmentation, transferability of pretrained face-voice embeddings to egocentric data, and availability of intra-clip identity annotations in Ego4D. The most consequential hand-made choice is the evaluation subset definition (Ego4D†), which directly determines the headline comparison. No new physical entities or formal axioms are introduced.

free parameters (6)

Self-Lifting cluster centroids = 50
Reduced from 1000 to 50 during finetuning to reflect Ego4D identity count; impacts embedding discriminability.
Transformer encoder depth = 1 layer
Single transformer layer chosen for temporal aggregation; no ablation shown.
Number of attention heads = 4 heads for encoder and cross-attention
Fixed hyperparameter; no sensitivity analysis.
Learning rate schedule = Adam, lr=1e-5, decay 0.2 every 5 epochs
Standard schedule; not ablated.
Utterance overlap threshold (train/dev) = 15% of groundtruth utterance duration
Used to filter segmentation outputs during training and development only; affects training distribution.
Evaluation subset (Ego4D†) = Only face tracks concurrent with detected utterances
Hand-chosen evaluation criterion that discards negative frames and defines the subset where SL-ASD exceeds baselines; central to the comparison in Table I.

assumptions (5)

domain assumption Each detected utterance contains exactly one speaker with no overlapping speech
Stated in Section II-B and Introduction; overlapping speech corrupts the audio branch embedding and propagates errors.
domain assumption Face tracks in egocentric video contain at least one frame with resolvable identity cues per utterance
The method's premise, stated in Section I: 'transient moments of clear facial visibility can suffice for robust associations.' If false, no face-voice match is possible.
domain assumption Pretrained face and speaker encoders (Inception-V1, ECAPA-TDNN) transfer to egocentric domains
Self-Lifting is frozen after finetuning on AVA-ActiveSpeaker and Ego4D; transfer quality determines matching accuracy.
domain assumption Ego4D inter-track identity annotations are consistent within video clips
Assumed in Section III-A to group all utterances of one identity in a clip during training; enables the supervised grouping protocol.
standard math Transformer self-attention and cross-attention are valid sequence-aggregation tools
Standard transformer machinery used without modification; no new math is introduced.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Face-Voice Association for Audiovisual Active Speaker Detection in Egocentric Recordings." pith.science (2026). https://pith.science/paper/VUT4VG2L

@misc{pith2026250618055,
  author       = {Pith},
  title        = {Pith review of: Face-Voice Association for Audiovisual Active Speaker Detection in Egocentric Recordings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VUT4VG2L}},
  note         = {Machine review of arXiv:2506.18055}
}

read the original abstract

Audiovisual active speaker detection (ASD) is conventionally performed by modelling the temporal synchronisation of acoustic and visual speech cues. In egocentric recordings, however, the efficacy of synchronisation-based methods is compromised by occlusions, motion blur, and adverse acoustic conditions. In this work, a novel framework is proposed that exclusively leverages cross-modal face-voice associations to determine speaker activity. An existing face-voice association model is integrated with a transformer-based encoder that aggregates facial identity information by dynamically weighting each frame based on its visual quality. This system is then coupled with a front-end utterance segmentation method, producing a complete ASD system. This work demonstrates that the proposed system, Self-Lifting for audiovisual active speaker detection(SL-ASD), achieves performance comparable to, and in certain cases exceeding, that of parameter-intensive synchronisation-based approaches with significantly fewer learnable parameters, thereby validating the feasibility of substituting strict audiovisual synchronisation modelling with flexible biometric associations in challenging egocentric scenarios.

Figures

Figures reproduced from arXiv: 2506.18055 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 32 canonical work pages

[1]

Active Speakers in Context,

J. L. Alcazar, F. C. Heilbron, L. Mai, F. Perazzi, J.-Y . Lee, P. Arbel ´aez, and B. Ghanem, “Active Speakers in Context,” 2020 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR) , 2020

work page 2020
[2]

Ava Active Speaker: An Audio-Visual Dataset for Active Speaker De- tection,

J. Roth, S. Chaudhuri, O. Klejch, R. Marvin, A. Gallagher, L. Kaver, S. Ramaswamy, A. Stopczynski, C. Schmid, Z. Xi, and C. Pantofaru, “Ava Active Speaker: An Audio-Visual Dataset for Active Speaker De- tection,” in Proc. Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2020

work page 2020
[3]

ASD-Transformer: Efficient Active Speaker Detection Using Self And Multimodal Transformers,

G. Datta, T. Etchart, V . Yadav, V . Hedau, P. Natarajan, and S.-F. Chang, “ASD-Transformer: Efficient Active Speaker Detection Using Self And Multimodal Transformers,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP) , 2022

work page 2022
[4]

Hello! My name is... Buffy

M. Everingham, J. Sivic, and A. Zisserman, “Hello! My name is... Buffy” – Automatic Naming of Characters in TV Video,” in British Machine Vision Conference, 2006

work page 2006
[5]

Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection,

R. Tao, Z. Pan, R. K. Das, X. Qian, M. Z. Shou, and H. Li, “Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection,” in Proc. 29th ACM Int. Conf. on Multimedia, 2021

work page 2021
[6]

A Light Weight Model for Active Speaker Detection,

J. Liao, H. Duan, K. Feng, W. Zhao, Y . Yang, and L. Chen, “A Light Weight Model for Active Speaker Detection,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , June 2023

work page 2023
[7]

How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild,

O. K ¨op¨ukl¨u, M. Taseska, and G. Rigoll, “How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , 2021

work page 2021
[8]

Rethinking audio- visual synchronization for active speaker detection,

A. Wuerkaixi, Y . Zhang, Z. Duan, and C. Zhang, “Rethinking audio- visual synchronization for active speaker detection,” in Proc. 32nd IEEE Int. Conf. on Machine Learning for Signal Processing , 2022

work page 2022

Show all 36 references

[9]

Look who’s talking: speaker detection using video and audio correlation,

R. Cutler and L. S. Davis, “Look who’s talking: speaker detection using video and audio correlation,” 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532) , vol. 3, 2000

2000
[10]

End-to-End Active Speaker Detection,

J. L. Alcazar, M. Cordes, C. Zhao, and B. Ghanem, “End-to-End Active Speaker Detection,” in European Conference on Computer Vision, 2022

2022
[11]

Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection,

K. Min, S. Roy, S. Tripathi, T. Guha, and S. Majumdar, “Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection,” in Euro. Conf. on Computer Vision , 2022

2022
[12]

MAAS: Multi-modal Assignation for Active Speaker Detection,

J. Le’on-Alc’azar, F. C. Heilbron, A. K. Thabet, and B. Ghanem, “MAAS: Multi-modal Assignation for Active Speaker Detection,” 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , 2021

2021
[13]

Improving Audiovisual Active Speaker Detection in Egocentric Recordings with the Data-Efficient Image Transformer,

J. Clarke, Y . Gotoh, and S. Goetze, “Improving Audiovisual Active Speaker Detection in Egocentric Recordings with the Data-Efficient Image Transformer,” in IEEE Automatic Speech Recognition and Un- derstanding Workshop (ASRU23), 2023

2023
[14]

Target Active Speaker Detection with Audio-visual Cues,

Y . Jiang, R. Tao, Z. Pan, and H. Li, “Target Active Speaker Detection with Audio-visual Cues,” in Proc. Interspeech, 2023

2023
[15]

Speaker Embedding Informed Audiovisual Active Speaker Detection for Egocentric Recordings,

J. Clarke, Y . Gotoh, and S. Goetze, “Speaker Embedding Informed Audiovisual Active Speaker Detection for Egocentric Recordings,” in Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP) , 2025

2025
[16]

Ego4D: Around the World in 3,000 Hours of Egocentric Video,

“Ego4D: Around the World in 3,000 Hours of Egocentric Video,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021

2022
[17]

Technical Report for Ego4D Long Term Action Anticipation Challenge 2023,

T. Ishibashi, K. Ono, N. Kugo, and Y . Sato, “Technical Report for Ego4D Long Term Action Anticipation Challenge 2023,” 2023. [Online]. Available: https://arxiv.org/abs/2307.01467

2023 arXiv
[18]

Seeking the shape of sound: An adaptive framework for learning voice-face association,

P. Wen, Q. Xu, Y . Jiang, Z. Yang, Y . He, and Q. Huang, “Seeking the shape of sound: An adaptive framework for learning voice-face association,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2021, pp. 16 347–16 356

2021
[19]

Self-lifting: A novel framework for unsupervised voice-face association learning,

G. Chen, D. Zhang, T. Liu, and X. Du, “Self-lifting: A novel framework for unsupervised voice-face association learning,” in Proceedings of the 2022 International Conference on Multimedia Retrieval , ser. ICMR ’22. New York, NY , USA: Association for Computing Machinery, 2022,...

2022
[20]

Learnable pins: Cross- modal embeddings for person identity,

A. Nagrani, S. Albanie, and A. Zisserman, “Learnable pins: Cross- modal embeddings for person identity,” in Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII . Berlin, Heidelberg: Springer- Verlag, 2018, p. 73–...

2018
[21]

Single-branch network for multimodal training,

M. S. Saeed, S. Nawaz, Yousaf, M. H. Khan, M. Z. Zaheer, K. Nan- dakumar, M. H. Yousaf, and A. Mahmood, “Single-branch network for multimodal training,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023

2023
[22]

Disentangled representation learning for cross-modal biometric matching,

H. Ning, X. Zheng, X. Lu, and Y . Yuan, “Disentangled representation learning for cross-modal biometric matching,” IEEE Transactions on Multimedia, vol. 24, pp. 1763–1774, 2022

2022
[23]

An efficient momentum framework for face- voice association learning,

Y . Qiu, Z. Yu, and Z. Gao, “An efficient momentum framework for face- voice association learning,” in Pattern Recognition and Computer Vision, Q. Liu, H. Wang, Z. Ma, W. Zheng, H. Zha, X. Chen, L. Wang, and R. Ji, Eds. Singapore: Springer Nature Singapore, 2024, pp. 271–283

2024
[24]

Audio-visual activity guided cross-modal identity association for active speaker detection,

R. Sharma and S. Narayanan, “Audio-visual activity guided cross-modal identity association for active speaker detection,” IEEE Open Journal of Signal Processing, vol. 4, pp. 225–232, 2023

2023
[25]

Favoa: Face-voice association favours ambiguous speaker detection,

H. Carneiro, C. Weber, and S. Wermter, “Favoa: Face-voice association favours ambiguous speaker detection,” in Artificial Neural Networks and Machine Learning – ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021...

2021
[26]

What in the world do we hear?: An ecological approach to auditory event perception,

W. Gaver, “What in the world do we hear?: An ecological approach to auditory event perception,” Ecological Psychology, vol. 5, pp. 1–29, 03 1993

1993
[27]

The development of infant learning about specific face–voice relations,

L. Bahrick, M. Hernandez-Reif, and R. Flom, “The development of infant learning about specific face–voice relations,” Developmental psy- chology, vol. 41, pp. 541–52, 05 2005

2005
[28]

Powerset multi-class cross entropy loss for neural speaker diarization,

A. Plaquet and H. Bredin, “Powerset multi-class cross entropy loss for neural speaker diarization,” in Proc. INTERSPEECH 2023 , 2023

2023
[29]

Going deeper with convolutions,

C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. E. Reed, D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 1–9, 2014. [Online]. Available: https://api.semanticsch...

2015
[30]

ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,

B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,” in Interspeech 2020 . ISCA, Oct 2020. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2020-2650

2020 doi
[31]

V oxCeleb: A Large-Scale Speaker Identification Dataset,

A. Nagrani, J. S. Chung, and A. Zisserman, “V oxCeleb: A Large-Scale Speaker Identification Dataset,” in Interspeech 2017. ISCA, Aug. 2017. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2017-950

2017 doi
[32]

Silero V AD: pre-trained enterprise-grade V oice Activity Detector (V AD), Number Detector and Language Classifier,

S. Team, “Silero V AD: pre-trained enterprise-grade V oice Activity Detector (V AD), Number Detector and Language Classifier,” https: //github.com/snakers4/silero-vad, 2024

2024
[33]

Speaker change detection using support vector machines,

V . Kartik, D. S. Satish, and C. C. Sekhar, “Speaker change detection using support vector machines,” in Proceedings of NOLISP , 2005, pp. 22–25

2005
[34]

Robust Object Recogni- tion Through Symbiotic Deep Learning In Mobile Robots,

J. Cartucho, R. Ventura, and M. Veloso, “Robust Object Recogni- tion Through Symbiotic Deep Learning In Mobile Robots,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018

2018
[35]

The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results,

M. Everingham, L. Van Gool, C. Williams, J. Winn, and A. Zis- serman, “The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results,” http://www.pascal-network.org/challenges/VOC/ voc2012/workshop/index.html

2012
[36]

LoCoNet: Long-Short Context Network for Active Speaker Detection,

X. Wang, F. Cheng, and G. Bertasius, “LoCoNet: Long-Short Context Network for Active Speaker Detection,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2024

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Active Speakers in Context,

J. L. Alcazar, F. C. Heilbron, L. Mai, F. Perazzi, J.-Y . Lee, P. Arbel ´aez, and B. Ghanem, “Active Speakers in Context,” 2020 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR) , 2020

work page 2020

[2] [2]

Ava Active Speaker: An Audio-Visual Dataset for Active Speaker De- tection,

J. Roth, S. Chaudhuri, O. Klejch, R. Marvin, A. Gallagher, L. Kaver, S. Ramaswamy, A. Stopczynski, C. Schmid, Z. Xi, and C. Pantofaru, “Ava Active Speaker: An Audio-Visual Dataset for Active Speaker De- tection,” in Proc. Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2020

work page 2020

[3] [3]

ASD-Transformer: Efficient Active Speaker Detection Using Self And Multimodal Transformers,

G. Datta, T. Etchart, V . Yadav, V . Hedau, P. Natarajan, and S.-F. Chang, “ASD-Transformer: Efficient Active Speaker Detection Using Self And Multimodal Transformers,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP) , 2022

work page 2022

[4] [4]

Hello! My name is... Buffy

M. Everingham, J. Sivic, and A. Zisserman, “Hello! My name is... Buffy” – Automatic Naming of Characters in TV Video,” in British Machine Vision Conference, 2006

work page 2006

[5] [5]

Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection,

R. Tao, Z. Pan, R. K. Das, X. Qian, M. Z. Shou, and H. Li, “Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection,” in Proc. 29th ACM Int. Conf. on Multimedia, 2021

work page 2021

[6] [6]

A Light Weight Model for Active Speaker Detection,

J. Liao, H. Duan, K. Feng, W. Zhao, Y . Yang, and L. Chen, “A Light Weight Model for Active Speaker Detection,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , June 2023

work page 2023

[7] [7]

How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild,

O. K ¨op¨ukl¨u, M. Taseska, and G. Rigoll, “How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , 2021

work page 2021

[8] [8]

Rethinking audio- visual synchronization for active speaker detection,

A. Wuerkaixi, Y . Zhang, Z. Duan, and C. Zhang, “Rethinking audio- visual synchronization for active speaker detection,” in Proc. 32nd IEEE Int. Conf. on Machine Learning for Signal Processing , 2022

work page 2022