{"id":"2a3a3f1d-33aa-41aa-8aeb-3b7b9f376191","arxiv_id":"2502.06012","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SCAN adds framewise voice comparison between reference speech and candidate audio to active speaker detection, improving mAP on Ego4D over the TalkNet and Light-ASD baselines.","lead":"This paper introduces SCAN, a module that compares a candidate speaker's known voice with the current audio to help egocentric cameras decide who is talking. It improves two existing active-speaker detectors by 14.5% and 10.3% relative mAP on the Ego4D benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reference speech may be sampled from the same video clip as the candidate audio, letting SCAN's cross-attention match identical waveforms rather than speaker identity; reported Ego4D gains could be inflated.","rationale":"Read in good faith, the paper proposes a plausible auxiliary module and shows consistent mAP improvements over two baselines on Ego4D and AVA. The frozen ECAPA encoder, public benchmarks, and self-supervised face finetuning are independent of the target metric, so I do not see circularity. The load-bearing condition is that reference speech AS is a clean enrollment of the candidate's voice, not a copy of the scored audio. The manuscript never states that AS is temporally disjoint from A, and the described library construction and random sampling leave open the possibility of exact waveform overlap. That would let Eq. (1)'s cross-attention solve active-speaker detection by acoustic self-match, which would not test SCAN's speaker-comparison mechanism. This is a concrete, testable concern rather than a settled flaw. I also note that the headline 14.5% and 10.3% improvements and the SOTA-bridging table use ground-truth identity-speech libraries, whereas the proposed self-supervised library yields smaller gains; this reinforces the need for careful evaluation reporting. The reader's CONDITIONAL verdict remains appropriate: acceptance should require the temporal-disjointness experiment, the hypothesized-library numbers, and ideally code release. If the experiment shows mAP collapse, the verdict should move to REJECT.","tokens_in":8374,"tokens_out":13234,"duration_ms":125908,"concrete_test":"Re-run the Ego4D evaluation with reference speech sampled under two conditions: (A) as reported, from the full identity-speech library; (B) with AS restricted to utterances temporally disjoint from the candidate track, e.g., taken from a different video clip or from a segment at least 5 seconds outside the track's temporal span. If TalkNet+SCAN mAP drops substantially in condition (B) (for example, from the reported 58.4 toward the 51.0 baseline), the headline improvement is partly a waveform-overlap shortcut rather than speaker-comparison reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SCAN's central claim depends on reference speech AS being a clean enrollment utterance of the candidate speaker, not the same acoustic segment being scored. Section II-C1 builds identity-speech libraries from 'all tracks in a given dataset' and attributes 'pre-diarised speech within the track's corresponding audio signal' to the identity. Section III-B then samples reference speech 'randomly selected from the relevant part of the identity-speech library' during training, with no stated temporal exclusion from the candidate audio A. If AS is drawn from the same video clip (or the same track) as A, then in Eq. (1) f_phi(AS) and the corresponding frames of f_phi(A) are embeddings of the identical waveform. The cross-attention softmax can peak at those frames, allowing the framewise classifier to predict activity by acoustic self-match rather than by speaker comparison. This would inflate the Ego4D mAP gains without exercising SCAN's intended mechanism. The paper does not specify whether enrollment and scoring audio are temporally disjoint, so this shortcut is not ruled out by the reported protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCAN, an auxiliary module for active speaker detection (ASD) that extracts speaker embeddings from both reference speech and the candidate audio signal, and compares them via a cross-attention mechanism. The authors also propose a self-supervised method for building identity-speech libraries by fine-tuning a face recognition model on video tracks with transformer layers and an impostor-frame classification objective. Experiments on Ego4D-AVD show that adding SCAN improves two speaker-embedding-naive baselines, TalkNet and Light-ASD, by 14.5% and 10.3% relative mAP when ground-truth identity-speech libraries are used, with smaller gains on AVA-ActiveSpeaker. The identity-speech library quality is evaluated using silhouette scores.","tokens_in":8553,"tokens_out":6703,"duration_ms":68543,"significance":"If the reported results hold, SCAN is a lightweight and modular contribution that can be attached to existing ASD systems and improves robustness in egocentric recordings. The use of a frozen ECAPA-TDNN encoder and evaluation on two public benchmarks are strengths. The idea of frame-level cross-attention between reference speech and candidate audio is interesting and distinct from prior speaker-embedding injection. However, the central claim is currently weakened by two issues: the paper does not rule out temporal overlap between reference speech and the audio being scored, and the evaluation lacks error bars or statistical tests. The identity-speech library evaluation is also thin, resting on a single silhouette score that the authors acknowledge shows only minor cluster separability. The contribution is therefore promising but not yet convincingly established.","major_comments":[{"comment":"The paper does not state whether reference speech A_S is temporally disjoint from the candidate audio A during training and evaluation. Since the identity-speech library is populated with pre-diarised speech from the tracks themselves (Section II-C1), A_S can be drawn from the same recording, or even the same waveform, as the audio window being scored. In that case, f_phi(A_S) and f_phi(A) are embeddings of identical acoustic content, and the cross-attention in Eq. (1) can peak at self-matching frames, allowing the classifier to rely on acoustic self-similarity rather than speaker identity. This is a load-bearing issue because it could inflate the Ego4D mAP gains in Table I. Please specify the temporal/excerpt exclusion rule used when sampling A_S, and add an experiment in which enrollment speech is guaranteed to come from a different recording or a non-overlapping time interval.","section":"Section III-B and Eq. (1)"},{"comment":"The claim that SCAN \"significantly improves\" the baselines is not supported by error bars, multiple runs, or significance tests. The AVA differences are small (for example, 93.9 vs 94.2 for Light-ASD with a ground-truth library), and without variability estimates it is not possible to tell which reported gaps are meaningful. Please report standard deviations over at least three seeds or otherwise provide a statistical comparison.","section":"Section IV-A and Table I"},{"comment":"The evaluation of the proposed identity-speech library method rests on a single silhouette score (0.16), which the authors themselves describe as \"only minor cluster separability\". This does not support the conclusion of a \"substantial improvement\" over TS-TalkNet's library generation. In particular, there is no downstream comparison of ASD mAP using the same baseline with the previous library versus the proposed library. Please add such a comparison, and consider reporting precision/recall of identity clustering against the pseudo-identity annotations mentioned in Section III-A.","section":"Section IV-B and Table III"}],"minor_comments":[{"comment":"The abstract and Table II report \"14.5% and 10.3%\" improvements without specifying that they correspond to ground-truth identity-speech libraries; with hypothesised libraries the relative gains are 13.7% and 5.2%. Please state the library condition explicitly.","section":"Abstract and Table II"},{"comment":"The static cosine similarity threshold (0.9) and the minimum reference-speech duration (2.5 s) appear to be ad hoc choices; a sensitivity analysis would make the identity-speech library method more reproducible and help assess its robustness.","section":"Section II-C1"},{"comment":"Validation-only evaluation is a standard limitation in ASD, but please state this in the abstract or conclusion so readers do not mistake the reported numbers for test-fold results.","section":"Section III-C"},{"comment":"Figure 4 lacks axis labels and a legend; adding them would make the similarity distributions interpretable.","section":"Figure 4"},{"comment":"Some table rows are hard to parse; for example, the row grouping in Table I is not fully clear from the header, and Table II cites SPELL via reference [28], which is a technical report on long-term action anticipation rather than an ASD system. Please check the citation and clarify the row semantics.","section":"Table I and Table II"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a multimedia journal, and the proposed module is architecturally sensible. The main risk is the unresolved temporal-overlap issue between reference speech and candidate audio, which could invalidate the headline Ego4D gains. If the authors can show that gains persist when enrollment speech is temporally disjoint, the paper would be significantly stronger. The lack of error bars is concerning but not fatal in this community; still, the authors should provide at least a multiple-seed comparison for the central table. I would not reject on the basis of validation-only results, as that is common practice in ASD, but the paper should be transparent about it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SCAN is a plausible auxiliary module that shows consistent gains on Ego4D, but the evaluation has an unaddressed protocol ambiguity that could inflate those gains. The novelty is real but incremental: framewise cross-attention between candidate audio and reference speech embeddings is a new combination for ASD, and the self-supervised impostor-frame face-recognition finetuning is a natural extension of TS-TalkNet. The paper does honest work — frozen ECAPA-TDNN, public benchmarks, both ground-truth and hypothesised identity libraries, and a modest AVA improvement that fits the argument that SCAN helps where vision is degraded. It also explicitly acknowledges the silhouette score of 0.16 as only minor cluster separability, which is in its favor.\n\nThe soft spots are the usual reproducibility issues: no code or data release, validation-fold results only, no error bars or significance tests, and a pseudo-identity annotation for the validation fold that is not released. More specific to this paper, the training protocol for sampling reference speech does not state that AS is temporally disjoint from the candidate audio A. Since the identity-speech library is built from 'pre-diarised speech within the track's corresponding audio signal,' reference speech can be drawn from the same video clip (indeed the same audio track) being scored. If that happens, the cross-attention in Eq. (1) can match identical waveforms, and the classifier learns an acoustic self-match rather than speaker identity. That would inflate the reported mAP gains on Ego4D. The paper should either enforce and state temporal exclusion or demonstrate that the gains persist when AS is guaranteed to come from a different recording.\n\nThat said, I don't think this is a load-bearing flaw in the architecture; it's a missing control in the protocol. The central idea is sound, and even if the gain shrinks, the module likely still helps. The paper deserves a serious referee, but the referee should ask for a temporal-disjointness control, error bars, and code or at least a detailed enrollment protocol. I'd send it to review rather than desk-reject.","headline":"SCAN is a plausible auxiliary module with consistent but unverified gains; the protocol must rule out identical-waveform matching before the numbers can be trusted.","tokens_in":9132,"tokens_out":2980,"would_cite":false,"duration_ms":25697,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that its SCAN module, which compares reference-speech and candidate-audio speaker embeddings through cross-attention, improves egocentric active speaker detection by 14.5% and 10.3% relative mAP over TalkNet and…","keywords":["active speaker detection","speaker embedding","egocentric video","cross-attention","self-supervised face recognition","identity-speech library","Ego4D","audiovisual fusion"],"falsifier":"Take the Ego4D validation clips where the candidate's face is occluded and another voice is active; if TalkNet+SCAN does not beat TalkNet on that subset, the specific disambiguation claim fails.","tokens_in":8126,"feed_emoji":"🎤","tokens_out":8727,"duration_ms":80626,"temperature":0.7,"pith_summary":"This paper tries to establish that active speaker detection on egocentric video becomes substantially more reliable when the system can compare the candidate speaker's reference voice against the current audio moment by moment. The proposed SCAN module extracts speaker embeddings from both pre-enrolled reference speech and the candidate audio track, then uses cross-attention to highlight when the current audio matches the reference voice, so frames with occluded faces or competing speakers can still be attributed correctly. Plugged into TalkNet and Light-ASD, SCAN raises mean average precision on the Ego4D-AVD validation fold by 14.5% and 10.3%, respectively, nearly matching larger recent models. The paper also contributes a self-supervised method that finetunes a face recognition model on video tracks to build identity-speech libraries robust to egocentric noise, replacing the prior enrollment approach. If these results hold, lightweight auxiliary speaker comparison could become a standard add-on for ASD systems deployed on wearable devices.","feed_headline":"Add-on network lifts egocentric active speaker detection by 14.5%","feed_subtitle":"Comparing each face's reference voice to the live audio disambiguates hard egocentric scenes.","key_machinery":"SCAN is a lightweight auxiliary network that, for each video frame, takes a one-second window of the candidate audio track and a clip of reference speech from the identity-speech library, embeds both with a frozen ECAPA-TDNN speaker verifier, and applies cross-attention so the audio window queries the reference clip. The attended output is classified as active or inactive and added as an auxiliary binary cross-entropy loss to the baseline ASD system. The second mechanism is a self-supervised face recognition finetune: an existing frame-based face model is wrapped with transformer encoder layers and trained to spot randomly inserted impostor frames within a face track, which teaches it to weight crisp identity-bearing frames and yields a track-level identity embedding for enrolling the speech library.","core_discovery":"The authors claim that the limiting factor for speaker-embedding-informed ASD is not just having reference speech, but actively comparing it with the candidate audio signal at frame level. SCAN does this by computing a cross-attention between embeddings of one-second audio windows from the test track and embeddings of reference speech from the identity-speech library, with the frozen ECAPA-TDNN speaker encoder. The resulting comparison signal is fed through a binary classifier as an auxiliary loss, and the output is shown to transfer to two very different baselines: TalkNet+SCAN reaches 58.4 mAP on Ego4D with a hypothesized library (58.0 without), and Light-ASD+SCAN reaches 59.9 (57.1 without); against the previous TS-TalkNet approach, TalkNet+SCAN is 4.4 points better with a hypothesized library. On AVA-ActiveSpeaker the gains are small, which the authors attribute to the rarity of visually occluded multi-talker scenes in exocentric footage. A second claimed contribution is that identity-speech libraries built with the self-supervised temporal face model are better than TS-TalkNet's enrollment, moving the cluster-separability score from -0.17 to 0.16 on Ego4D validation.","pith_inferences":["The cross-attention comparison could be applied to any ASD backbone with an audio encoder, and its benefit should scale with the degree of visual corruption; a natural next test is a dataset built specifically from wearable cameras with head motion.","Because SCAN's reference audio is drawn from a pre-enrolled library, the method is compatible with a streaming setting where reference speech is accumulated after the first confirmed utterance, turning ASD into a bootstrapping process.","The reported cluster-separability score of 0.16 suggests the enrollment model still confuses some identities; improving temporal face clustering, or adding a speaker-verification filter on enrolled speech, would likely push the Ego4D mAP above the reported numbers.","One implication the authors do not draw out: SCAN's framewise comparison output could be reused as a weak diarization signal, since high attention between candidate audio and reference speech marks who is speaking at each moment."],"forward_implications":["ASD systems that ignore speaker identity can be upgraded with a small auxiliary module rather than redesigned, since SCAN is architecture-agnostic.","On egocentric benchmarks, speaker-embedding-informed ASD can approach the performance of much larger specialized models: Light-ASD+SCAN reaches 59.9 mAP on Ego4D versus 60.7 for the current listed top system.","The benefit concentrates in exactly the hard cases of visual occlusion and overlapping speech, so gains on exocentric data should stay small, as observed on AVA-ActiveSpeaker.","Identity-speech library quality is a bottleneck: better enrollment directly raises the ceiling, as shown by the gap between ground-truth and hypothesized libraries for TalkNet+SCAN (58.4 vs 58.0) and Light-ASD+SCAN (59.9 vs 57.1)."],"supporting_citations":[{"why":"Supplies the TalkNet baseline that SCAN is attached to and improves by 14.5% relative mAP on Ego4D.","marker":"[1]"},{"why":"Supplies TS-TalkNet, the previous speaker-embedding-informed system and the identity-speech library protocol that SCAN extends and outperforms.","marker":"[7]"},{"why":"Supplies the Light-ASD baseline, the second system SCAN is integrated with, and the false-positive example motivating speaker comparison.","marker":"[13]"},{"why":"Supplies ECAPA-TDNN, the frozen speaker embedding extractor used for both reference speech and candidate audio in SCAN.","marker":"[16]"},{"why":"Supplies VoxCeleb, the dataset used to pretrain the speaker-recognition model behind SCAN's embeddings.","marker":"[19]"},{"why":"Supplies Ego4D-AVD, the egocentric benchmark where the main mAP gains are measured.","marker":"[10]"},{"why":"Supplies AVA-ActiveSpeaker, the exocentric benchmark used to show that SCAN's gains are scenario-specific.","marker":"[8]"},{"why":"Supplies the Partial FC face recognition model that is finetuned with the self-supervised temporal objective for identity-speech library enrollment.","marker":"[24]"},{"why":"Supplies the transformer encoder layers used to give the face model temporal context for the self-supervised finetuning.","marker":"[25]"}],"fun_headline_variants":["Reference-voice comparison lifts egocentric ASD by 14.5%","SCAN: speaker-embedding comparison disambiguates egocentric ASD","Compare reference and live voices to disambiguate egocentric ASD","Speaker-embedding comparison boosts egocentric ASD by 14.5%","SCAN lifts TalkNet and Light-ASD on Ego4D by 14.5% and 10.3%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed gains depend on the stored reference clips for each face actually being that person's voice, and the paper's own evaluation reports only minor separability between same-person and different-person clips (a score of 0.16), so enrollment noise is the point of fragility.","fun_headline_variants_meta":{"raw":{"variants":["Reference-voice comparison lifts egocentric ASD by 14.5%","SCAN: speaker-embedding comparison disambiguates egocentric ASD","Compare reference and live voices to disambiguate egocentric ASD","Speaker-embedding comparison boosts egocentric ASD by 14.5%","SCAN lifts TalkNet and Light-ASD on Ego4D by 14.5% and 10.3%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001608,"raw_usage":{"total_tokens":6431,"prompt_tokens":1000,"completion_tokens":5431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":5323}},"tokens_in":616,"tokens_out":5431,"duration_ms":36280,"temperature":1.0,"reasoning_tokens":5323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:01:58.624764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the Ego4D validation clips where the candidate's face is occluded and another voice is active; if TalkNet+SCAN does not beat TalkNet on that subset, the specific disambiguation claim fails.","supporting_citations":[{"cited_title":"Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the TalkNet baseline that SCAN is attached to and improves by 14.5% relative mAP on Ego4D."},{"cited_title":"Partial FC: Training 10 Million Identities on a Single Ma- chine,","cited_arxiv_id":null,"evidence_quote":"Supplies the Partial FC face recognition model that is finetuned with the self-supervised temporal objective for identity-speech library enrollment."},{"cited_title":"Attention is All you Need,","cited_arxiv_id":null,"evidence_quote":"Supplies the transformer encoder layers used to give the face model temporal context for the self-supervised finetuning."}],"review_version":1}