Pith. sign in

REVIEW 4 major objections 5 minor 57 references

CueNet: Robust Audio-Visual Speaker Extraction through Cross-Modal Cue Mining and Interaction

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Audio-visual speaker extraction can tolerate badly corrupted video if the model separates three kinds of visual cues and adaptively trusts the ones still reliable.

desk verdict A credible incremental advance in robust AV speaker extraction, with a clean cue-disentanglement story, but the 'no degraded training' claim is slightly overstated given the AV-HuBERT teacher's masking pretraining. read the letter →

arxiv 2603.01530 v2 pith:CBYMFXHG submitted 2026-03-02 cs.MM

Jiadong Wang , Ke Zhang , Xinyuan Qian , Ruijie Tao , Haizhou Li , Björn Schuller This is my paper
classification cs.MM
keywords audio-visualspeakerextractionvisualdegradationrobustnesscuedisentanglementacousticsynchronisationsemanticcross-modalinteractionreliability-basedfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that an audio-visual speaker extraction system can stay accurate when the video signal is badly degraded, even though it was never trained on degraded footage. The trick is to stop treating the visual input as one blob and instead disentangle it into three complementary cues: speaker identity, acoustic synchronisation (lip motion matching pitch and energy), and semantic synchronisation (lip motion matching linguistic content). A reliability-estimation module then weights each cue per frame and per feature dimension, so the model leans on whichever cues are still trustworthy. On two benchmarks, the proposed system outperforms prior methods across four types of 50% visual corruption—blur, concealment, missing feature blocks, and face disappearance—while also improving cross-dataset generalisation.

What carries the argument

The load-bearing machinery is the three-cue decomposition of the visual stream. A low-level learner produces the speaker cue and the acoustic-synchronisation cue; a high-level learner produces the semantic-synchronisation cue. Both learners use interaction blocks that update speech and visual features through gated cross-modal attention. The acoustic and semantic cues are supervised by k-means token clusters derived from pitch/spectrogram features and from a self-supervised audio-visual model respectively. The cue interaction module then estimates reliability for each cue and fuses the three cue-enhanced speech representations with attention weights computed along feature and temporal dimens

What would settle it

Take a model trained exactly as in the paper and evaluate it on a set of real-world degraded videos with natural occlusion, motion blur, or face disappearances that persist longer than the 5-frame blocks used in the paper. If its SI-SNRi advantage over prior systems shrinks or reverses compared with the simulated-corruption results, the central claim of degradation-agnostic robustness is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that robust audio-visual speaker extraction does not require seeing degraded videos during training; it requires decomposing the visual modality into three functional cues and dynamically fusing them by reliability. A hierarchical audio-visual learner interacts lip-movement features with the mixed-audio representation to produce a speaker embedding, an acoustic-synchronisation representation, and a semantic-synchronisation representation, each supervised by its own discrete token targets. A cue interaction module then estimates the reliability of each cue and fuses cue-enhanced speech features with attention weights over feature and temporal dimensions. Across LRS3 a

Load-bearing premise

The robustness evidence relies on test-time corruptions that block 50% of frames in 5-frame chunks using four hand-picked degradation types; if real-world visual impairments have different statistics—say occlusions lasting seconds or desynchronised audio-video tracks—the claimed advantage may not survive.

Editorial extensions

If this is right

  • The method keeps higher extraction quality than existing systems when 50% of the video frames are corrupted, without any degraded video in training.
  • Disentangling the visual signal into speaker, acoustic, and semantic cues improves robustness more than treating the visual embedding as a single entity.
  • Adaptive reliability-weighted fusion outperforms simple concatenation of cues, and the gap widens as degradation gets more severe.
  • Training on one dataset and testing on another shows larger relative gains, suggesting the cue decomposition helps domain shift, not just same-domain corruption.
  • The framework is proposed for speaker extraction but the authors suggest it could transfer to other audio-visual tasks such as speech recognition and active speaker detection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reliability weights truly track cue trustworthiness, the model's attention maps could be used as a diagnostic to identify which visual information the model is ignoring when video is corrupted.
  • The k-means token supervision is a coarse proxy; using continuous or richer phonetic targets for the semantic cue might further improve robustness under severe degradation.
  • The claimed robustness is demonstrated on short corruption blocks (5 frames); real-world occlusions or dropouts that persist for seconds may stress the reliability estimator in ways the current experiments do not capture.
  • The approach could be paired with audio-only fallback mechanisms for the face-missing case, potentially making the system degrade gracefully as visual information goes to zero.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CueNet, an audio-visual speaker extraction model that aims to remain robust to visual degradation without training on degraded videos. The model disentangles three cues—speaker information, acoustic synchronisation, and semantic synchronisation—through a hierarchical audio-visual learner, and combines them via a cue interaction module that estimates cue reliability and dynamically fuses the resulting speech representations. Experiments on LRS3 and VoxCeleb2, including a cross-domain setting, report improved SI-SNRi, SDR, and PESQ over several baselines under four synthetic visual degradations (Gaussian Blur, Concealment, Masked Feature, Face Missing). Ablation studies support the individual contributions of the three cues and the interaction module. The central claim is that the model achieves robustness without relying on degraded visual data during training, with the semantic synchronisation cue being trained on pseudo-labels derived from AV-HuBERT features.

Significance. If the central claim holds, the work addresses an important and underexplored problem in audio-visual speaker extraction: robustness to imperfect visual inputs in a way that does not require enumerating degradation types during training. The three-cue decomposition is conceptually motivated and the dynamic fusion mechanism is a reasonable contribution. The ablation studies, cross-domain evaluation, and comparison with several recent baselines are valuable. However, the significance is substantially tempered by the fact that the semantic cue is supervised by AV-HuBERT, a teacher pre-trained with masked multimodal prediction, which introduces an indirect pathway for the model to learn robustness to masked visual inputs. This undercuts the 'no degraded training' claim unless the teacher pathway is shown to be non-essential. The empirical evaluation also lacks error bars and significance testing, and some baselines are missing degraded-condition results.

major comments (4)
  1. [§III-B, Eq. (3)] The semantic synchronisation cue C_w is trained to match pseudo-labels Y_w obtained by k-means clustering of AV-HuBERT features. AV-HuBERT (Ref. [42]) is pre-trained with masked multimodal cluster prediction, i.e., it is optimized to be robust to masked visual inputs. Thus C_w is distilled from a teacher that has seen degraded (masked) video during pre-training, even though CueNet itself is not fine-tuned on degraded videos. This is a direct pathway by which the 'without relying on degraded videos during training' claim (Abstract, Sec. I, Sec. II-C) is weakened, and it may explain the large gains under Masked Feature and Face Missing in Tables I and III. The paper neither acknowledges this nor ablates the teacher. Please add an ablation that replaces AV-HuBERT labels with a clean-trained visual speech recognition or audio-based semantic teacher and report the robustness contributions; al
  2. [§IV-A, Tables I-III] All results are reported without error bars or significance tests, and appear to come from a single run. Several comparisons are very close (e.g., VoxCeleb2 Clean SI-SNRi: CueNet-fast 13.7 vs IIANet 13.6 in Table II; SDR for CueNet-fast is lower than IIANet under some conditions). Without repeated runs or variance information, the claim that CueNet 'consistently outperforms' all baselines is not fully supported. Please report mean±std over multiple seeds or at least the number of runs and a significance test on the key comparisons.
  3. [§IV-B, Table II] Several baselines are missing degraded-condition results entirely (VisualVoice, ConvTasNet, AV-DPRNN, Muse, AV-Sep are shown only under Clean). The conclusion that the method outperforms 'state-of-the-art methods across various types of visual degradation' cannot be verified for these methods. Please either evaluate them under the same degradation protocol or explicitly restrict the conclusion to the methods for which degraded numbers are reported.
  4. [§IV-A, §V] The robustness evaluation is based on a single hand-designed protocol: 50% of frames corrupted in blocks of 5 consecutive frames, for four degradation types. Real-world visual impairments (e.g., long occlusions, desynchronisation, compression) may have different statistics. The 'strong robustness ... for real-world' claim in the conclusion goes beyond the evidence. Please add experiments that vary the corruption ratio and block length, or test on a real-world corrupted data set; otherwise the external validity of the robustness claim is limited.
minor comments (5)
  1. [§IV-C, Fig. 5 caption] Caption typo: 'Maksed feature' should be 'Masked feature'.
  2. [Table II] In the PESQ column for AV-TFGridNet, '2.432.45' is missing a space; fix formatting.
  3. [§III-B] There is a grammatical error: 'The main goal is to while gradually transforming video to three distinct cues' — a verb or phrase is missing. Please rephrase.
  4. [Intro, Ref. [15]] The text cites 'Grant and Seitz [15]' but the reference list contains only K. W. Grant. Please add the co-author or correct the citation.
  5. [§IV-D, Fig. 4] The caption says 'Attention cross the temporal dimension'; should be 'across'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CueNet's robustness claims rest on independent empirical comparisons, not on a self-referential derivation.

full rationale

CueNet is an empirical systems paper, not a derivation. The central claim—outperforming baselines under unseen visual degradations—is supported by direct comparisons on LRS3 and VoxCeleb2 under a held-out degradation protocol (Sec. IV-B), not by a derivation from its own definitions. The three cues are auxiliary training targets: speaker ID, k-means tokens over pitch/spectrogram features, and k-means tokens over AV-HuBERT features (Sec. III-B, Eq. 3) supervise intermediate representations. These are not re-used as evaluation metrics, so no prediction is forced by construction. The only overlapping-author citations (SeaNet backend [37], MOMuSe/Memo [30], [31], and prior tracking/ASR works [2], [3]) are component reuse or related work; none is invoked as a uniqueness theorem or used to forbid alternatives. The use of AV-HuBERT pseudo-labels is an external, independently trained model, and the possible inheritance of masking robustness from AV-HuBERT's masked multimodal pretraining is a legitimate attribution/confound concern for the 'no degraded training' narrative, but it is not a circular step in the paper's derivation chain. Ablations and fusion comparisons provide independent empirical validation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central robustness claim is empirical and rests primarily on the chosen cue supervision (k-means clusters), the sufficiency of the three-cue decomposition, and the synthetic degradation protocol used for evaluation. No independent evidence is provided that the cues remain disentangled under unseen corruption, so these assumptions are load-bearing.

free parameters (4)
  • K_acoustic (k-means cluster count for acoustic cue) = 512
    Supervision tokens Y_a for acoustic synchronisation are produced by k-means on pitch/spectrogram features of the clean target speech (Sec. III-B). The paper calls the choice empirical and ablates K from 64 to 512 in Table VI.
  • K_semantic (k-means cluster count for semantic cue) = 48
    Y_w tokens from AV-HuBERT features are clustered into 48 classes to roughly match English phonemes (Sec. IV-C). No systematic ablation reported.
  • Loss weighting coefficients = unspecified ('appropriate weighting')
    Sec. III-D says all five losses are 'combined with appropriate weighting' without giving coefficients; central training balance is not reproducible.
  • Degradation evaluation protocol (50% corruption, 5-frame blocks) = 50% / 5 frames
    Sec. IV-A sets the test-time corruption level and pattern. The robustness claim is measured under this protocol; results at other severities appear only in Fig. 5 for Masked Feature.
assumptions (4)
  • domain assumption Three-cue decomposition (speaker identity, acoustic synchronisation, semantic synchronisation) is a valid and sufficient decomposition of visual guidance for speaker extraction.
    Sec. II-C and III-B assume these cues capture the useful information in the visual modality; if a fourth type of visual information matters, the robustness mechanism may miss it.
  • domain assumption K-means token supervision on clean target speech and AV-HuBERT features validly isolates the intended cues.
    Y_a is derived from pitch/spectrogram of the clean target signal S (Sec. III-B) and Y_w from AV-HuBERT features (Sec. III-B); the paper provides no analysis that these tokens actually encode synchronisation rather than other properties.
  • domain assumption Findings from human perception (McGurk effect, Grant & Seitz mouth-opening/F2 correlation) transfer analogically to a deep network.
    Introduction and Sec. II-C motivate the cue split by human listening; the paper gives no evidence that networks should exhibit the same cue robustness.
  • domain assumption The face video V is temporally synchronized with the target speech S and contains the target speaker.
    Sec. III-A defines V as 'temporally synchronized with S'; the method cannot correct desynchronised or wrong-speaker videos.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CueNet: Robust Audio-Visual Speaker Extraction through Cross-Modal Cue Mining and Interaction." pith.science (2026). https://pith.science/paper/CBYMFXHG

@misc{pith2026260301530,
  author       = {Pith},
  title        = {Pith review of: CueNet: Robust Audio-Visual Speaker Extraction through Cross-Modal Cue Mining and Interaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CBYMFXHG}},
  note         = {Machine review of arXiv:2603.01530}
}
read the original abstract

Audio-visual speaker extraction has attracted increasing attention, as it removes the need for pre-registered speech and leverages the visual modality as a complement to audio. Although existing methods have achieved impressive performance, the issue of degraded visual inputs has received relatively little attention, despite being common in real-world scenarios. Previous attempts to address this problem have mainly involved training with degraded visual data. However, visual degradation can occur in many unpredictable ways, making it impractical to simulate all possible cases during training. In this paper, we aim to enhance the robustness of audio-visual speaker extraction against impaired visual inputs without relying on degraded videos during training. Inspired by observations from human perceptual mechanisms, we propose an audio-visual learner that disentangles speaker information, acoustic synchronisation, and semantic synchronisation as distinct cues. Furthermore, we design a dedicated interaction module that effectively integrates these cues to provide a reliable guidance signal for speaker extraction. Extensive experiments demonstrate the strong robustness of the proposed model under various visual degradations and its clear superiority over existing methods.

Figures

Figures reproduced from arXiv: 2603.01530 by the authors.

Figure 1
Figure 1. Pipeline of CueNet. The audio-visual learner performs cross-modal interaction between the audio feature [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 4
Figure 4. Attention cross the temporal dimension. There are 50 temporal frames. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Performance across different degree of corrpution. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 7 linked inside Pith

  1. [42]

    Learning audio- visual speech representation by masked multimodal cluster prediction,

    B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio- visual speech representation by masked multimodal cluster prediction,” inInternational Conference on Learning Representations, 2022

    2022

  2. [1]

    Late audio-visual fusion for in-the-wild speaker diarization,

    Z. Pan, G. Wichern, F. G. Germain, A. Subramanian, and J. Le Roux, “Late audio-visual fusion for in-the-wild speaker diarization,” in2024 IEEE International Conference on Acoustics, Speech, and Signal Pro- cessing Workshops (ICASSPW). IEEE, 2024, pp. 174–178

    2024

  3. [2]

    Predict-and-update network: Audio-visual speech recognition inspired by human speech perception,

    J. Wang, X. Qian, and H. Li, “Predict-and-update network: Audio-visual speech recognition inspired by human speech perception,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  4. [3]

    Audio-visual cross- attention network for robotic speaker tracking,

    X. Qian, Z. Wang, J. Wang, G. Guan, and H. Li, “Audio-visual cross- attention network for robotic speaker tracking,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 550–562, 2022

    2022

  5. [4]

    My lips are concealed: Audio-visual speech enhancement through obstructions,

    T. Afouras, J. S. Chung, and A. Zisserman, “My lips are concealed: Audio-visual speech enhancement through obstructions,” inProceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol. 2019, 2019, pp. 4295–4299

    2019

  6. [5]

    Multimodal attention fusion for target speaker extraction,

    H. Sato, T. Ochiai, K. Kinoshita, M. Delcroix, T. Nakatani, and S. Araki, “Multimodal attention fusion for target speaker extraction,” in2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 778–784

    2021

  7. [6]

    Time-domain audio-visual speech separation on low quality videos,

    Y . Wu, C. Li, J. Bai, Z. Wu, and Y . Qian, “Time-domain audio-visual speech separation on low quality videos,” inICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 256–260

    2022

  8. [7]

    A two-stage audio-visual speech separation method without visual signals for testing and tuples loss with dynamic margin,

    Y . Liu, Y . Deng, and Y . Wei, “A two-stage audio-visual speech separation method without visual signals for testing and tuples loss with dynamic margin,”IEEE Journal of Selected Topics in Signal Processing, 2024

    2024

Show all 57 references
  1. [8]

    Ravss: Robust audio- visual speech separation in multi-speaker scenarios with missing visual cues,

    T. Pan, J. Liu, B. Wang, J. Tang, and G. Wu, “Ravss: Robust audio- visual speech separation in multi-speaker scenarios with missing visual cues,” inProceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 4748–4756

    2024

  2. [9]

    Multi-modal multi-correlation learning for audio-visual speech separation,

    X. Wang, X. Kong, X. Peng, and Y . Lu, “Multi-modal multi-correlation learning for audio-visual speech separation,” inProc. Interspeech 2022, 2022, pp. 886–890

    2022

  3. [10]

    Visualvoice: Audio-visual speech separa- tion with cross-modal consistency,

    R. Gao and K. Grauman, “Visualvoice: Audio-visual speech separa- tion with cross-modal consistency,” in2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021, pp. 15 490–15 500

    2021

  4. [11]

    Explaining face-voice matching decisions: The contribution of mouth movements, stimulus effects and response biases,

    N. Lavan, H. Smith, L. Jiang, and C. McGettigan, “Explaining face-voice matching decisions: The contribution of mouth movements, stimulus effects and response biases,”Attention, Perception, & Psychophysics, vol. 83, no. 5, pp. 2205–2216, 2021. JOURNAL OF LATEX CLASS FILES, VOL...

    2021

  5. [12]

    Rethinking the visual cues in audio-visual speaker extraction,

    J. Li, M. Ge, Z. Pan, R. Cao, L. Wang, J. Dang, and S. Zhang, “Rethinking the visual cues in audio-visual speaker extraction,” inProc. Interspeech 2023, 2023, pp. 3754–3758

    2023

  6. [13]

    Muse: Multi-modal target speaker extraction with visual cues,

    Z. Pan, R. Tao, C. Xu, and H. Li, “Muse: Multi-modal target speaker extraction with visual cues,” inICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6678–6682

    2021

  7. [14]

    Hearing lips and seeing voices,

    H. McGurk and J. MacDonald, “Hearing lips and seeing voices,”Nature, vol. 264, no. 5588, pp. 746–748, 1976

    1976

  8. [15]

    The effect of speechreading on masked detection thresh- olds for filtered speech,

    K. W. Grant, “The effect of speechreading on masked detection thresh- olds for filtered speech,”The Journal of the Acoustical Society of America, vol. 109, no. 5, pp. 2272–2275, 2001

    2001

  9. [16]

    Neural target speech extraction: An overview,

    K. Zmolikova, M. Delcroix, T. Ochiai, K. Kinoshita, J. ˇCernock`y, and D. Yu, “Neural target speech extraction: An overview,”IEEE Signal Processing Magazine, vol. 40, no. 3, pp. 8–29, 2023

    2023

  10. [17]

    Multi-cue guided semi-supervised learning toward target speaker separation in real environments,

    J. Xu, J. Cui, Y . Hao, and B. Xu, “Multi-cue guided semi-supervised learning toward target speaker separation in real environments,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 151–163, 2023

    2023

  11. [18]

    Multi- level speaker representation for target speaker extraction,

    K. Zhang, J. Li, S. Wang, Y . Wei, Y . Wang, Y . Wang, and H. Li, “Multi- level speaker representation for target speaker extraction,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  12. [19]

    Usef-tse: Universal speaker embedding free target speaker extraction,

    B. Zeng and M. Li, “Usef-tse: Universal speaker embedding free target speaker extraction,”IEEE Transactions on Audio, Speech and Language Processing, 2025

    2025

  13. [20]

    Contextual speech extraction: Leveraging textual history as an implicit cue for target speech extraction,

    M. Kim, R. Mira, H. Chen, S. Petridis, and M. Pantic, “Contextual speech extraction: Leveraging textual history as an implicit cue for target speech extraction,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  14. [21]

    Conceptbeam: Concept driven target speech extraction,

    Y . Ohishi, M. Delcroix, T. Ochiai, S. Araki, D. Takeuchi, D. Niizumi, A. Kimura, N. Harada, and K. Kashino, “Conceptbeam: Concept driven target speech extraction,” inProceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 4252–4260

    2022

  15. [22]

    Sense: Semantic-aware high-fidelity universal speech enhancement,

    X. Li, H. Xie, Z. Wang, Z. Zhang, L. Xiao, and L. Xie, “Sense: Semantic-aware high-fidelity universal speech enhancement,”arXiv preprint arXiv:2509.24708, 2025

    2025 arXiv

  16. [23]

    Av-crossnet: An audiovisual complex spectral mapping network for speech separation by leveraging narrow-and cross-band modeling,

    V . A. Kalkhorani, C. Yu, A. Kumar, K. Tan, B. Xu, and D. Wang, “Av-crossnet: An audiovisual complex spectral mapping network for speech separation by leveraging narrow-and cross-band modeling,”IEEE Journal of Selected Topics in Signal Processing, 2025

    2025

  17. [24]

    Audio-visual speech separation and dereverberation with a two-stage multimodal network,

    K. Tan, Y . Xu, S.-X. Zhang, M. Yu, and D. Yu, “Audio-visual speech separation and dereverberation with a two-stage multimodal network,” IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 3, pp. 542–553, 2020

    2020

  18. [25]

    Spatialnet: Extensively learning spatial information for multichannel joint speech separation, denoising and dereverberation,

    C. Quan and X. Li, “Spatialnet: Extensively learning spatial information for multichannel joint speech separation, denoising and dereverberation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 1310–1323, 2024

    2024

  19. [26]

    Fasnet: Low- latency adaptive beamforming for multi-microphone audio processing,

    Y . Luo, C. Han, N. Mesgarani, E. Ceolini, and S.-C. Liu, “Fasnet: Low- latency adaptive beamforming for multi-microphone audio processing,” in2019 IEEE automatic speech recognition and understanding work- shop (ASRU). IEEE, 2019, pp. 260–267

    2019

  20. [27]

    Multi-modal multi-channel target speech separation,

    R. Gu, S.-X. Zhang, Y . Xu, L. Chen, Y . Zou, and D. Yu, “Multi-modal multi-channel target speech separation,”IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 3, pp. 530–541, 2020

    2020

  21. [28]

    An overview of deep-learning-based audio-visual speech en- hancement and separation,

    D. Michelsanti, Z.-H. Tan, S.-X. Zhang, Y . Xu, M. Yu, D. Yu, and J. Jensen, “An overview of deep-learning-based audio-visual speech en- hancement and separation,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1368–1396, 2021

    2021

  22. [29]

    Unified audio visual cues for target speaker extraction,

    T. Wu, S. He, J. Pan, H. Huang, Z. Mo, and X. Zhang, “Unified audio visual cues for target speaker extraction,” inProc. Interspeech 2024, 2024, pp. 4343–4347

    2024

  23. [30]

    Momuse: Momentum multi-modal target speaker extraction for real-time scenarios with impaired visual cues,

    J. Li, K. Zhang, S. Wang, K. A. Lee, M.-W. Mak, and H. Li, “Momuse: Momentum multi-modal target speaker extraction for real-time scenarios with impaired visual cues,”arXiv preprint arXiv:2412.08247, 2024

    2024 arXiv

  24. [31]

    Memo: Attentional momentum for real-time audio-visual speaker extraction under impaired visual conditions,

    J. Li, W. Wu, S. Wang, Z. Pan, K. A. Lee, H. Meng, and H. Li, “Memo: Attentional momentum for real-time audio-visual speaker extraction under impaired visual conditions,”arXiv preprint arXiv:2507.15294, 2025

    2025 arXiv

  25. [32]

    Tf-gridnet: Making time-frequency domain models great again for monaural speaker separation,

    Z.-Q. Wang, S. Cornell, S. Choi, Y . Lee, B.-Y . Kim, and S. Watanabe, “Tf-gridnet: Making time-frequency domain models great again for monaural speaker separation,” inICASSP 2023-2023 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 202...

    2023

  26. [33]

    Separate in the speech chain: Cross-modal conditional audio-visual target speech extraction,

    Z. Mu and X. Yang, “Separate in the speech chain: Cross-modal conditional audio-visual target speech extraction,”arXiv preprint arXiv:2404.12725, 2024

    2024 arXiv

  27. [34]

    Clearervoice-studio: Bridging advanced speech processing research and practical deployment,

    S. Zhao, Z. Pan, and B. Ma, “Clearervoice-studio: Bridging advanced speech processing research and practical deployment,”arXiv preprint arXiv:2506.19398, 2025

    2025 arXiv

  28. [35]

    Av-sepformer: Cross-attention sepformer for audio-visual target speaker extraction,

    J. Lin, X. Cai, H. Dinkel, J. Chen, Z. Yan, Y . Wang, J. Zhang, Z. Wu, Y . Wang, and H. Meng, “Av-sepformer: Cross-attention sepformer for audio-visual target speaker extraction,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICAS...

    2023

  29. [36]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  30. [37]

    Audio-visual target speaker extraction with selective auditory attention,

    R. Tao, X. Qian, Y . Jiang, J. Li, J. Wang, and H. Li, “Audio-visual target speaker extraction with selective auditory attention,”IEEE Transactions on Audio, Speech and Language Processing, 2025

    2025

  31. [38]

    Pitch range variations improve cognitive processing of audio messages,

    E. Rodero, R. F. Potter, and P. Prieto, “Pitch range variations improve cognitive processing of audio messages,”Human communication re- search, vol. 43, no. 3, pp. 397–413, 2017

    2017

  32. [39]

    Intonation and speaker identification,

    E. Abberton and A. J. Fourcin, “Intonation and speaker identification,” Language and Speech, vol. 21, no. 4, pp. 305–318, 1978

    1978

  33. [40]

    Dual-path rnn: efficient long sequence modeling for time-domain single-channel speech separation,

    Y . Luo, Z. Chen, and T. Yoshioka, “Dual-path rnn: efficient long sequence modeling for time-domain single-channel speech separation,” inICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 46–50

    2020

  34. [41]

    Time domain audio visual speech separation,

    J. Wu, Y . Xu, S.-X. Zhang, L.-W. Chen, M. Yu, L. Xie, and D. Yu, “Time domain audio visual speech separation,” in2019 IEEE automatic speech recognition and understanding workshop (ASRU). IEEE, 2019, pp. 667–673

    2019

  35. [43]

    Rethinking the inception architecture for computer vision,

    C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826

    2016

  36. [44]

    Sdr–half-baked or well done?

    J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “Sdr–half-baked or well done?” inICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 626–630

    2019

  37. [45]

    Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Model,

    H. Martel, J. Richter, K. Li, X. Hu, and T. Gerkmann, “Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Model,” inProc. INTERSPEECH 2023, 2023, pp. 1673–1677

    2023

  38. [46]

    An audio-visual speech separation model inspired by cortico-thalamo-cortical circuits,

    K. Li, F. Xie, H. Chen, K. Yuan, and X. Hu, “An audio-visual speech separation model inspired by cortico-thalamo-cortical circuits,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 10, pp. 6637–6651, 2024

    2024

  39. [47]

    Iianet: an intra-and inter-modality attention network for audio-visual speech separation,

    K. Li, R. Yang, F. Sun, and X. Hu, “Iianet: an intra-and inter-modality attention network for audio-visual speech separation,” inProceedings of the 41st International Conference on Machine Learning, 2024, pp. 29 181–29 200

    2024

  40. [48]

    Deep audio-visual speech recognition,

    T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep audio-visual speech recognition,”IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 12, pp. 8717–8727, 2018

    2018

  41. [49]

    V oxceleb2: Deep speaker recognition,

    J. S. Chung, A. Nagrani, and A. Zisserman, “V oxceleb2: Deep speaker recognition,”arXiv preprint arXiv:1806.05622, 2018

    2018 arXiv

  42. [50]

    Looking into your speech: Learning cross-modal affinity for audio-visual speech separation,

    J. Lee, S.-W. Chung, S. Kim, H.-G. Kang, and K. Sohn, “Looking into your speech: Learning cross-modal affinity for audio-visual speech separation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1336–1345

    2021

  43. [51]

    Watch or listen: Robust audio- visual speech recognition with visual corruption modeling and reliability scoring,

    J. Hong, M. Kim, J. Choi, and Y . M. Ro, “Watch or listen: Robust audio- visual speech recognition with visual corruption modeling and reliability scoring,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 18 783–18 794

    2023

  44. [52]

    Restoring speaking lips from occlusion for audio-visual speech recognition,

    J. Wang, Z. Pan, M. Zhang, R. T. Tan, and H. Li, “Restoring speaking lips from occlusion for audio-visual speech recognition,” inProceedings of the AAAI conference on artificial intelligence, vol. 38, no. 17, 2024, pp. 19 144–19 152

    2024

  45. [53]

    Delving into high-quality synthetic face occlusion segmentation datasets,

    K. T. V oo, L. Jiang, and C. C. Loy, “Delving into high-quality synthetic face occlusion segmentation datasets,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4711–4720

    2022

  46. [54]

    Data2vec: A general framework for self-supervised learning in speech, vision and language,

    A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” inInternational conference on machine learning. PMLR, 2022, pp. 1298–1312. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGU...

    2022

  47. [55]

    Adam: A method for stochastic optimization,

    D. P. Kingma, “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  48. [56]

    Long short-term memory,

    A. Graves, “Long short-term memory,”Supervised sequence labelling with recurrent neural networks, pp. 37–45, 2012

    2012

  49. [57]

    How many phonemes does the english language have,

    A. L. Bizzocchi, “How many phonemes does the english language have,” International Journal on Studies in English Language and Literature (IJSELL), vol. 5, no. 10, pp. 36–46, 2017

    2017

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.