Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

MeMo: Attentional Momentum for Real-time Audio-visual Speaker Extraction under Impaired Visual Conditions

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MeMo claims that a real-time audio-visual speaker extractor can keep focusing on a target speaker after the face video becomes missing, occluded, or blurry, by storing past extracted speech in two adaptive memory banks that supply…

desk verdict Solid empirical step for real-time AV-TSE, but the >2 dB headline is conditioned on a clean visual first glimpse and the paper should say so. read the letter →

arxiv 2507.15294 v2 pith:6LUQVJRA submitted 2025-07-21 cs.SD cs.MM

Junjie Li , Wenxuan Wu , Shuai Wang , Zexu Pan , Kong Aik Lee , Helen Meng , Haizhou Li This is my paper
classification cs.SDcs.MM
keywords audio-visualtargetspeakerextractionattentionalmomentumadaptivememorybankself-enrollmentreal-timestreamingvisualimpairmentpseudo-autoregressivetraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MeMo claims that real-time audio-visual target speaker extraction can keep separating a chosen speaker even after the face video becomes missing, occluded, or blurry, by carrying attention forward through memory instead of relying on the current frame. The system stores two kinds of reference information from speech it has already extracted in earlier processing windows: speaker identity embeddings and contextual speech embeddings. At each new window, attention-based retrieval pulls the most relevant stored cues and feeds them to the extractor alongside the current, possibly impaired, visual input. On the paper's streaming evaluations, the contextual-bank version improves SI-SNR by more than 2 dB over the baseline under impaired visuals, from 8.13 dB to 10.34 dB with the TDSE backbone, and similar gains appear with two other backbones. If correct, a brief clean glimpse of a speaker can anchor continued extraction during long periods of visual failure, a key step toward practical deployment.

What carries the argument

The load-bearing object is the adaptive memory bank, specifically the contextual bank. Attentional momentum names the mechanism: once attention is established from a visual glimpse, the model keeps its focus by storing and retrieving reference information from its own previously extracted speech. The speaker bank holds speaker embeddings and retrieves a fused identity cue by self-attention over its slots; the contextual bank holds speech embeddings of past windows and uses two cross-attention steps to retrieve the temporal pattern most relevant to the current mixture. Storing, retrieval, and updating, by first-in-first-out or attention-based selection, define the momentum loop, and a pseudo-autoregressive training objective with curriculum weighting lets the system learn to use its own imperfect output as reference.

What would settle it

Take the trained contextual-bank model and evaluate it online with increasing fractions of impaired visual frames after a clean initialization. The paper's claim predicts that its SI-SNR stays nearly flat around 10 dB while the visual-only baseline drops below 7 dB; a result where the two curves converge as impairment grows would falsify the attentional-momentum mechanism.

Watch

Extended reading notes

Core claim

The central claim is that attentional momentum can be implemented in an audio-visual target speaker extraction system by two adaptive memory banks that are populated with the system's own past output, a self-enrollment strategy. The speaker bank stores per-window speaker embeddings and retrieves the most relevant identity cue by self-attention over its slots. The contextual bank stores speech embeddings of previously extracted windows and retrieves temporal context through two cross-attention layers, one selecting content within each slot and one weighting slots relative to the current mixture. The paper reports that the contextual bank, used alone, yields substantial performance gains over the baseline, particularly under impaired visual conditions in the online mode, with improvements exceeding 2 dB in SI-SNR (System 4: 10.34 dB versus 8.13 dB for TDSE under impaired visuals), while adding only small parameter and computation overhead, with a real-time factor around 0.11 on the GPU used. A pseudo-autoregressive training strategy with curriculum learning from ground-truth to self-enrolled speech is what makes the two-stage extraction trainable. The gain is not tied to one architecture: the contextual bank also improves the USEV and BSRNN backbones.

Load-bearing premise

The reported gains assume the very first processing window sees visually clean frames of the target speaker, because that first glimpse produces the self-enrolled speech that seeds both memory banks; with an impaired initialization window, the best system's SI-SNR falls from 10.34 dB to 8.91 dB.

Editorial extensions

If this is right

  • A visual-only AV-TSE system that loses the face video mid-conversation can keep extracting the target speaker from the memory of past extracted speech, narrowing the gap to offline results.
  • The contextual bank transfers to different separation backbones, so the design is an add-on component rather than a new end-to-end model.
  • Performance stays nearly flat as the fraction of impaired visual frames grows, and stays stable on utterances up to 40 seconds, where the visual-only baseline degrades steadily.
  • In speaker-switching conversations, the system recovers after the switch without an explicit bank reset, though emptying the bank does not help.
  • The real-time factor remains low enough for live use, and the extra parameters are small, making the mechanism compatible with streaming deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The clean-initialization requirement means the practical deployment rule is to obtain one clean frame at startup or bootstrap from pre-enrolled speech; the paper's results do not support starting with an already impaired camera.
  • Because the latest contextual embedding is the most informative slot and receives the highest attention weight in about 95% of cases, a simpler single-slot or exponentially decaying memory might capture most of the momentum benefit as a testable simplification.
  • The failure of transformer-based backbones to train stably under the two-stage modality shift suggests the distribution change from visual-only to visual-plus-self-enrolled input is the real obstacle, and gradually mixing the modalities during training may unlock attention-based extractors.
  • The attention-based selection update helps the speaker bank but not the contextual bank, consistent with low-level contextual embeddings being hard to score; a hard rule that always keeps the newest contextual slot could be compared against attention-based selection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MeMo, a framework for real-time audio-visual target speaker extraction that augments standard AV-TSE models with two adaptive memory banks—a speaker bank storing identity embeddings and a contextual bank storing historical speech embeddings. Through a self-enrollment mechanism, outputs from previous sliding-window steps are re-encoded and retrieved via attention as supplementary references, enabling extraction to continue when current visual cues are impaired. The framework is evaluated on VoxCeleb2-derived mixtures with three visual impairment types (missing, lip concealment, low resolution), in both offline and online modes, and is applied to three backbones (TDSE, USEV, BSRNN). The main reported results are SI-SNR improvements exceeding 2 dB over the TDSE baseline under impaired visual conditions in online mode, and roughly 27% relative improvement, with real-time factors around 0.11 on a V100 GPU. The paper also introduces a speaker-switching evaluation set.

Significance. If the reported results are reproducible, MeMo offers a practical and architecture-agnostic way to compensate for degraded or missing visual cues in streaming AV-TSE, which is an important deployment concern. The paper's strengths include evaluation across multiple backbones, a clear ablation of memory-bank designs (FIFO vs. attention-based selection, slot count, self-enrollment length), analysis of window length/shift and loss weight, a long-utterance robustness study, a speaker-switching test set, and released code/demos. The main weaknesses are the absence of statistical variability measures, an internal inconsistency in the BSRNN results in Table XIII, and a load-bearing initialization assumption that is underemphasized in the abstract and Section V-B.

major comments (4)
  1. [Table XIII] The row for MeMo (BSRNN) reports SI-SNR = 10.98 dB but SDR = 9.47 dB, meaning SDR is 1.51 dB lower than SI-SNR. In every other row of the same table, SDR is higher than SI-SNR (for example, TDSE has SI-SNR 8.13 and SDR 8.53). Since SDR and SI-SNR are closely related metrics and the discrepancy is unique to one system, this strongly suggests an error in the reported values or in the evaluation pipeline. This row is used to support the claim of consistent gains across backbones, so the inconsistency must be resolved and the table corrected.
  2. [Section V-D] The headline claim in Section V-B that the contextual bank yields 'improvements exceeding 2 dB in SI-SNR' is obtained under the initialization protocol stated in Section IV-C2, where 'clean visual frames without impairments were used' in the 0-th window step. Table IX shows that if the initialization window is itself visually impaired, System 4 drops from 10.34 dB to 8.91 dB SI-SNR, reducing the gain over the TDSE baseline (8.13 dB) from 2.21 dB to 0.78 dB. The abstract and Section V-B present the >2 dB improvement without this qualifier, which overstates the method's robustness under fully impaired visual input. The authors should either restrict the claim to the clean-initialization condition or present the Table IX result prominently as a limitation.
  3. [Tables III, V, XIII, XIV] All reported results appear to come from single training runs with no error bars, confidence intervals, or multi-seed variance. Several comparisons in the ablations are small (e.g., 8.64 vs. 8.66 dB in Table IV), and without variance information it is difficult to judge whether these differences are meaningful. Adding results over at least 3 seeds with mean and standard deviation, or at least reporting the variance for the key comparisons in Tables III and V, would materially strengthen the empirical claims.
  4. [Section V-H] The paper states that Transformer-based backbones (SEANet, IIANet, Sepformer) 'could not be optimized stably' and provides no quantitative results for them. This is relevant to the paper's claim that MeMo is a generalizable framework applicable to various models. As written, the generalization claim is supported only for CNN- and RNN-based backbones. The authors should either include the failed attempts with details (e.g., training curves or loss divergence) in a supplementary file, or explicitly narrow the generality claim in the abstract and conclusion to the architectures that were successfully trained.
minor comments (5)
  1. [Section III-E] The text repeatedly uses 'Cause' where 'Because' is intended (e.g., 'Cause we use SI-SNR as the training loss' and 'Cause we use L_SI-SNR as loss function'). Please correct these typographical errors.
  2. [Fig. 8 caption] The caption says 'the impaired ratio of whole utterance is lessly higher than 60%'. This phrasing is unclear and ungrammatical. Please rephrase, e.g., 'since the initialization window is clean, the effective impaired ratio of the whole utterance is less than 60%'.
  3. [Table II header] The header 'Evalation FormulationSettings' is malformed and should be split into 'Evaluation Mode' and 'Formulation / Settings'.
  4. [Table XIII] The column header 'Macs (M) (G/s)' mixes units without clear labeling. Please clarify the units for computational complexity, e.g., 'MACs (M)' and 'FLOPs (G/s)' or similar.
  5. [Throughout] There are spacing/formatting inconsistencies such as 'A V-TSE', 'V oxCeleb2', and 'lessly' in Fig. 8. A careful proofreading pass is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MeMo's gains are empirical comparisons against external baselines; the clean-initialization caveat is an explicit boundary condition, not a circular step.

full rationale

This is an empirical systems paper, not a derivation-to-fit paper. The central claim (Section V-B: 'improvements exceeding 2 dB in SI-SNR') is supported by direct comparisons in Tables III, V, VIII, and XIII against external baselines such as TDSE, USEV, BSRNN, ImagineNET, MuSE, and MoMuSE under the same evaluation protocol. No equation in the paper is fitted to the quantity it is then said to predict; Equations (3)-(20) are definitions of attention operations, loss, and normalization, and Equation (21) is an energy-smoothing rule, none of which reduces to the reported SI-SNR improvements. The only self-citation is MoMuSE [20], which is explicitly disclosed as prior work by the same group and used both as motivation and as a comparison baseline; it is not invoked as an authority to forbid alternatives or to force the framework's design. The paper also contains an explicit boundary condition in Section IV-C2 ('In the 0-th window step, clean visual frames without impairments were used') and a self-reported limitation in Section V-D/Table IX showing that impaired initialization reduces System 4 from 10.34 dB to 8.91 dB SI-SNR. This qualifies the 2 dB claim and is a legitimate deployment caveat, but it is not circularity: the claimed result is not defined in terms of the clean-initialization condition, and the sensitivity is empirically measured rather than assumed. The PAR training strategy and curriculum learning are standard stabilization techniques adopted from cited work and are not used as evidence for the headline performance gain. Overall, the paper is self-contained against external benchmarks, and no load-bearing step reduces to its own inputs by construction.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities; the memory banks are software constructs whose behavior is fully specified by the model equations. The main unverified assumptions are the reliability of self-enrolled speech and the availability of a clean visual initialization.

free parameters (6)
  • loss weight beta = 0.2
    Balances Stage 1 and Stage 2 losses in Eq. 19; chosen via the ablation in Fig. 9.
  • normalization weight gamma = 0.7
    Energy normalization weight in Eq. 21; set by hand to control output loudness.
  • memory slots N (inference) = 1
    Default number of slots per bank; Tables IV and VI show that larger N degrades performance.
  • window length T_win = 2s
    Sliding window length; Table X shows 4s gives higher SI-SNR but higher RTF.
  • shift length T_sh = 0.2s
    Sliding shift length; Table XI shows a performance peak at 0.2s.
  • curriculum epochs ep_cr = 50
    Number of epochs over which the curriculum weight alpha increases from 0 to 1.
assumptions (4)
  • domain assumption Self-enrolled estimated speech from previous windows contains reliable speaker identity and contextual acoustic information.
    Central premise of MeMo; if previous outputs are too noisy or wrong, errors accumulate, which the paper tries to mitigate with curriculum learning and energy normalization.
  • domain assumption A clean visual frame is available for the 0-th window step in the deployed scenario.
    The online inference strategy in Algorithm 1 and Table IX rely on clean visual initialization; with impaired initialization, gains shrink from 10.34 to 8.91 dB for System 4.
  • domain assumption The pre-trained ECAPA-TDNN speaker encoder from WeSpeaker provides useful embeddings for self-enrolled speech.
    The speaker bank depends on this frozen external encoder trained on VoxCeleb; no adaptation is performed for the impaired-visual domain.
  • standard math Standard attention softmax operations and concatenation-based fusion are sufficient to combine reference cues.
    The architecture uses standard attention equations (Eqs. 3-17) from [63] as unproved background.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MeMo: Attentional Momentum for Real-time Audio-visual Speaker Extraction under Impaired Visual Conditions." pith.science (2026). https://pith.science/paper/6LUQVJRA

@misc{pith2026250715294,
  author       = {Pith},
  title        = {Pith review of: MeMo: Attentional Momentum for Real-time Audio-visual Speaker Extraction under Impaired Visual Conditions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6LUQVJRA}},
  note         = {Machine review of arXiv:2507.15294}
}
read the original abstract

Audio-visual Target Speaker Extraction (AV-TSE) aims to isolate a target speaker's voice from multi-speaker environments by leveraging visual cues as guidance. However, the performance of AV-TSE systems heavily relies on the quality of these visual cues. In extreme scenarios where visual cues are missing or severely degraded, the system may fail to accurately extract the target speaker. In contrast, humans can maintain attention on a target speaker even in the absence of explicit auxiliary information. Motivated by such human cognitive ability, we propose a novel framework called MeMo, which incorporates two adaptive memory banks to store attention-related information. MeMo is specifically designed for real-time scenarios: once initial attention is established, the system maintains attentional momentum over time, even when visual cues become unavailable. We conduct comprehensive experiments to verify the effectiveness of MeMo. Experimental results demonstrate that our proposed framework achieves SI-SNR improvements of at least 2 dB over the corresponding baseline.

Figures

Figures reproduced from arXiv: 2507.15294 by the authors.

Figure 1
Figure 1. Typical AV-TSE relies solely on visual cues, but [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of MeMo framework. MeMo consists of two types of adaptive memory banks: speaker bank and contextual [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Illustration of retrieval operation from contextual bank. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (6 more)
Figure 3
Figure 3. Figure 3: Illustration of retrieval operation from speaker bank. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 5
Figure 5. Figure 5: The basic idea of attentional momentum. y(k), v(k) and xˆ(k) denote the speech mixture, visual cues and estimated speech at the k-th online window step. the core concept of attentional momentum. At the start of online processing, MeMo initially relies on visual cues v(…
Figure 6
Figure 6. Figure 6: Examples of visual impairments. initialization, where only the visual cue v is used without the pre-enrolled speech, is referred to as VInit (applied to both the speaker bank and contextual bank). Stage 2: We utilize the self-enrolled estimated speech xˆ 1 as reference…
Figure 7
Figure 7. Figure 7: Attention shifts from one target speaker to another at [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: The performance of different systems on different [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Study for the effect of hyper-parameter β in the loss function. 3) Study on Loss Function Weight: As shown in [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CueNet: Robust Audio-Visual Speaker Extraction through Cross-Modal Cue Mining and Interaction

    cs.MM 2026-03 conditional novelty 6.0 of 10

    CueNet disentangles speaker, acoustic-synchronisation, and semantic-synchronisation cues from audio-visual speech and fuses them by estimated reliability, preserving extraction performance under visual degradation wit...

Reference graph

Works this paper leans on

82 extracted references · 72 canonical work pages · cited by 1 Pith paper

  1. [1]

    Some experiments on the recognition of speech, with one and with two ears,

    E. C. Cherry, “Some experiments on the recognition of speech, with one and with two ears,”The Journal of the acoustical society of America, vol. 25, no. 5, pp. 975–979, 1953

    1953

  2. [2]

    The cocktail party phenomenon: A review of research on speech intelligibility in multiple-talker conditions,

    A. W. Bronkhorst, “The cocktail party phenomenon: A review of research on speech intelligibility in multiple-talker conditions,”Acta acustica united with acustica, vol. 86, no. 1, pp. 117–128, 2000

    2000

  3. [3]

    Seeing to hear better: evidence for early audio-visual interactions in speech identification,

    J.-L. Schwartz, F. Berthommier, and C. Savariaux, “Seeing to hear better: evidence for early audio-visual interactions in speech identification,” Cognition, vol. 93, no. 2, pp. B69–B78, 2004

    work page 2004

  4. [4]

    Visual contribution to speech intelligibility in noise,

    W. H. Sumby and I. Pollack, “Visual contribution to speech intelligibility in noise,”The journal of the acoustical society of america, vol. 26, no. 2, pp. 212–215, 1954

    work page 1954

  5. [5]

    Muse: Multi-modal target speaker extraction with visual cues,

    Z. Pan, R. Tao, C. Xu, and H. Li, “Muse: Multi-modal target speaker extraction with visual cues,” inICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 6678–6682

    work page 2021

  6. [6]

    Usev: Universal speaker extraction with visual cue,

    Z. Pan, M. Ge, and H. Li, “Usev: Universal speaker extraction with visual cue,”IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 30, p. 3032–3045, sep 2022. 13

    work page 2022

  7. [7]

    Time domain audio visual speech separation,

    J. Wu, Y . Xu, S.-X. Zhang, L.-W. Chen, M. Yu, L. Xie, and D. Yu, “Time domain audio visual speech separation,” in2019 IEEE automatic speech recognition and understanding workshop (ASRU). IEEE, 2019, pp. 667–673

    work page 2019

  8. [8]

    Neural target speech extraction: An overview,

    K. Zmolikova, M. Delcroix, T. Ochiai, K. Kinoshita, J. ˇCernock´y, and D. Yu, “Neural target speech extraction: An overview,”IEEE Signal Processing Magazine, vol. 40, no. 3, pp. 8–29, 2023

    2023

Show all 82 references
  1. [9]

    An overview of deep-learning-based audio-visual speech en- hancement and separation,

    D. Michelsanti, Z.-H. Tan, S.-X. Zhang, Y . Xu, M. Yu, D. Yu, and J. Jensen, “An overview of deep-learning-based audio-visual speech en- hancement and separation,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1368–1396, 2021

    2021

  2. [10]

    PIA VE: A Pose-Invariant Audio- Visual Speaker Extraction Network,

    Q. Liu, M. Ge, Z. Wu, and H. Li, “PIA VE: A Pose-Invariant Audio- Visual Speaker Extraction Network,” inProc. INTERSPEECH 2023, 2023, pp. 3719–3723

    2023

  3. [11]

    Speaker extraction with co-speech gestures cue,

    Z. Pan, X. Qian, and H. Li, “Speaker extraction with co-speech gestures cue,”IEEE Signal Processing Letters, vol. 29, pp. 1467–1471, 2022

    2022

  4. [12]

    Rethinking the Visual Cues in Audio-Visual Speaker Extraction,

    J. Li, M. Ge, Z. Pan, R. Cao, L. Wang, J. Dang, and S. Zhang, “Rethinking the Visual Cues in Audio-Visual Speaker Extraction,” in Proc. INTERSPEECH 2023, 2023, pp. 3754–3758

    2023

  5. [13]

    FaceFilter: Audio- Visual Speech Separation Using Still Images,

    S.-W. Chung, S. Choe, J. S. Chung, and H.-G. Kang, “FaceFilter: Audio- Visual Speech Separation Using Still Images,” inProc. Interspeech 2020, 2020, pp. 3481–3485

    2020

  6. [14]

    c 2av-tse: Context and confidence-aware audio visual target speaker extraction,

    W. Wu, X. Chen, S. Wang, J. Wang, L. Meng, X. Wu, H. Meng, and H. Li, “c 2av-tse: Context and confidence-aware audio visual target speaker extraction,”IEEE Journal of Selected Topics in Signal Process- ing 2025

    2025

  7. [15]

    Incorporating linguistic constraints from external knowledge source for audio-visual target speech extraction,

    W. Wu, S. Wang, X. Wu, H. Meng, and H. Li, “Incorporating linguistic constraints from external knowledge source for audio-visual target speech extraction,”Interspeech 2025

    2025

  8. [16]

    Iianet: An intra- and inter-modality attention network for audio-visual speech separation,

    K. Li, R. Yang, F. Sun, and X. Hu, “Iianet: An intra- and inter-modality attention network for audio-visual speech separation,” inInternational Conference on Machine Learning, 2024

    2024

  9. [17]

    Hearing lips in noise: Universal viseme-phoneme mapping and transfer for robust audio-visual speech recognition,

    Y . Hu, R. Li, C. Chen, C. Qin, Q.-S. Zhu, and E. S. Chng, “Hearing lips in noise: Universal viseme-phoneme mapping and transfer for robust audio-visual speech recognition,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023

    2023

  10. [18]

    Diarization is hard: Some experiences and lessons learned for the jhu team in the inaugural dihard challenge

    G. Sell, D. Snyder, A. McCree, D. Garcia-Romero, J. Villalba, M. Ma- ciejewski, V . Manohar, N. Dehak, D. Povey, S. Watanabeet al., “Diarization is hard: Some experiences and lessons learned for the jhu team in the inaugural dihard challenge.” inInterspeech, 2018, pp. 2808– 2812

    2018

  11. [19]

    Noise- disentanglement metric learning for robust speaker verification,

    Y . Sun, H. Zhang, L. Wang, K. A. Lee, M. Liu, and J. Dang, “Noise- disentanglement metric learning for robust speaker verification,” in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

    2023

  12. [20]

    Momuse: Momentum multi-modal target speaker extraction for real-time scenarios with impaired visual cues,

    J. Li, K. Zhang, S. Wang, K. A. Lee, M.-W. Mak, and H. Li, “Momuse: Momentum multi-modal target speaker extraction for real-time scenarios with impaired visual cues,” in2025 IEEE International Conference on Multimedia and Expo (ICME), 2025, pp. 1–6

    2025

  13. [21]

    Ravss: Robust audio- visual speech separation in multi-speaker scenarios with missing visual cues,

    T. Pan, J. Liu, B. Wang, J. Tang, and G. Wu, “Ravss: Robust audio- visual speech separation in multi-speaker scenarios with missing visual cues,” inProceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 4748–4756

    2024

  14. [22]

    Switching variational auto- encoders for noise-agnostic audio-visual speech enhancement,

    M. Sadeghi and X. Alameda-Pineda, “Switching variational auto- encoders for noise-agnostic audio-visual speech enhancement,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

    2021

  15. [23]

    Robust unsupervised audio-visual speech enhance- ment using a mixture of variational autoencoders,

    X.-P. M.Sadeghi, “Robust unsupervised audio-visual speech enhance- ment using a mixture of variational autoencoders,” inICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7534–7538

    2020

  16. [24]

    Time-domain audio-visual speech separation on low quality videos,

    Y . Wu, C. Li, J. Bai, Z. Wu, and Y . Qian, “Time-domain audio-visual speech separation on low quality videos,” inICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 256–260

    2022

  17. [25]

    Imaginenet: Target speaker extraction with intermittent visual cue through embedding inpainting,

    Z. Pan, W. Wang, M. Borsdorf, and H. Li, “Imaginenet: Target speaker extraction with intermittent visual cue through embedding inpainting,” inICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

    2023

  18. [26]

    My lips are concealed: Audio-visual speech enhancement through obstructions,

    T. Afouras, J. S. Chung, and A. Zisserman, “My lips are concealed: Audio-visual speech enhancement through obstructions,” inInterspeech, 2019

    2019

  19. [27]

    Multi-cue guided semi-supervised learning toward target speaker separation in real environments,

    J. Xu, J. Cui, Y . Hao, and B. Xu, “Multi-cue guided semi-supervised learning toward target speaker separation in real environments,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023

    2023

  20. [28]

    A two-stage audio-visual speech separation method without visual signals for testing and tuples loss with dynamic margin,

    Y . Liu, Y . Deng, and Y . Wei, “A two-stage audio-visual speech separation method without visual signals for testing and tuples loss with dynamic margin,”IEEE Journal of Selected Topics in Signal Processing, vol. 18, no. 3, pp. 459–472, 2024

    2024

  21. [29]

    Cross-modal speech separation without visual information during testing,

    Y . Y .Liu and Y .Wei, “Cross-modal speech separation without visual information during testing,” in2023 IEEE Biomedical Circuits and Systems Conference (BioCAS). IEEE, 2023, pp. 1–5

    2023

  22. [30]

    Robust audio-visual speech enhancement: Correcting misassignments in complex environments with advanced post-processing,

    W. Ren, K.-H. Hung, R. Chao, Y . Li, H.-M. Wang, and Y . Tsao, “Robust audio-visual speech enhancement: Correcting misassignments in complex environments with advanced post-processing,” in2024 27th Conference of the Oriental COCOSDA International Committee for the Co-ordinatio...

    2024

  23. [31]

    Cocktail party listening in a dynamic multitalker environment,

    D. S. Brungart and B. D. Simpson, “Cocktail party listening in a dynamic multitalker environment,”Perception & psychophysics, vol. 69, no. 1, pp. 79–91, 2007

    2007

  24. [32]

    The advantage of knowing where to listen,

    G. Kidd, T. L. Arbogast, C. R. Mason, and F. J. Gallun, “The advantage of knowing where to listen,”The Journal of the Acoustical Society of America, vol. 118, no. 6, pp. 3804–3815, 2005

    2005

  25. [33]

    Object continuity enhances selective auditory attention,

    V . Best, E. J. Ozmeral, N. Kopˇco, and B. G. Shinn-Cunningham, “Object continuity enhances selective auditory attention,”Proceedings of the National Academy of Sciences, vol. 105, no. 35, pp. 13 174–13 178, 2008

    2008

  26. [34]

    Enhanced learning through multimodal training: evidence from a comprehensive cognitive, physical fitness, and neuroscience intervention,

    N. Ward, E. Paul, P. Watson, G. Cooke, C. Hillman, N. J. Cohen, A. F. Kramer, and A. K. Barbey, “Enhanced learning through multimodal training: evidence from a comprehensive cognitive, physical fitness, and neuroscience intervention,”Scientific reports, vol. 7, no. 1, p. 5808, 2017

    2017

  27. [35]

    The important role of contex- tual information in speech perception in cochlear implant users and its consequences in speech tests,

    J. G. Dingemanse and A. Goedegebure, “The important role of contex- tual information in speech perception in cochlear implant users and its consequences in speech tests,”Trends in Hearing

  28. [36]

    Attention and working memory in human auditory cortex,

    B. Barton and A. A. Brewer, “Attention and working memory in human auditory cortex,” inThe Human Auditory System, S. Hatzopoulos, A. Ciorba, and P. H. Skarzynski, Eds. Rijeka: IntechOpen, 2019, ch. 1

    2019

  29. [37]

    Interactions between attention and working memory,

    E. Awh, E. K. V ogel, and S.-H. Oh, “Interactions between attention and working memory,”Neuroscience, vol. 139, no. 1, pp. 201–208, 2006

    2006

  30. [38]

    Look once to hear: Target speech hearing with noisy examples,

    B. Veluri, M. Itani, T. Chen, T. Yoshioka, and S. Gollakota, “Look once to hear: Target speech hearing with noisy examples,” ser. CHI ’24. New York, NY , USA: Association for Computing Machinery, 2024

    2024

  31. [39]

    Multimodal attention fusion for target speaker extraction,

    H. Sato, T. Ochiai, K. Kinoshita, M. Delcroix, T. Nakatani, and S. Araki, “Multimodal attention fusion for target speaker extraction,” in2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 778–784

    2021

  32. [40]

    Multimodal speakerbeam: Single channel target speech extraction with audio-visual speaker clues

    T. Ochiai, M. Delcroix, K. Kinoshita, A. Ogawa, and T. Nakatani, “Multimodal speakerbeam: Single channel target speech extraction with audio-visual speaker clues.” inINTERSPEECH, 2019, pp. 2718–2722

    2019

  33. [41]

    Memory networks,

    J. Weston, S. Chopra, and A. Bordes, “Memory networks,”International Conference on Learning Representations, ICLR 2015

    2015

  34. [42]

    End-to-end memory net- works,

    S. Sukhbaatar, J. Weston, R. Ferguset al., “End-to-end memory net- works,”Advances in neural information processing systems, vol. 28, 2015

    2015

  35. [43]

    Unsupervised feature learning via non-parametric instance discrimination,

    Z. Wu, Y . Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3733– 3742

    2018

  36. [44]

    Cromm-vsr: Cross-modal memory augmented visual speech recognition,

    M. Kim, J. Hong, S. J. Park, and Y . M. Ro, “Cromm-vsr: Cross-modal memory augmented visual speech recognition,”IEEE Transactions on Multimedia, vol. 24, pp. 4342–4355, 2022

    2022

  37. [45]

    Multi-temporal lip-audio memory for visual speech recognition,

    J. H. Yeo, M. Kim, and Y . M. Ro, “Multi-temporal lip-audio memory for visual speech recognition,” inICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

    2023

  38. [46]

    Multi-modality associative bridging through memory: Speech sound recollected from face video,

    M. Kim, J. Hong, S. J. Park, and Y . M. Ro, “Multi-modality associative bridging through memory: Speech sound recollected from face video,” 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 296–306, 2021

    2021

  39. [47]

    Akvsr: Audio knowledge empowered visual speech recognition by compressing audio knowledge of a pretrained model,

    J. H. Yeo, M. Kim, J. Choi, D. H. Kim, and Y . M. Ro, “Akvsr: Audio knowledge empowered visual speech recognition by compressing audio knowledge of a pretrained model,”IEEE Transactions on Multimedia, 2024

    2024

  40. [48]

    Distinguishing homophenes using multi-head visual-audio memory for lip reading,

    M. Kim, J. H. Yeo, and Y . M. Ro, “Distinguishing homophenes using multi-head visual-audio memory for lip reading,” inAAAI Conference on Artificial Intelligence, 2022

    2022

  41. [49]

    Speech reconstruction with reminiscent sound via visual voice memory,

    J. Hong, M. Kim, S. J. Park, and Y . M. Ro, “Speech reconstruction with reminiscent sound via visual voice memory,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3654–3667, 2021. 14

    2021

  42. [50]

    Modeling attention and memory for auditory selection in a cocktail party environment,

    J. Xu, J. Shi, G. Liu, X. Chen, and B. Xu, “Modeling attention and memory for auditory selection in a cocktail party environment,” inAAAI Conference on Artificial Intelligence, 2018

    2018

  43. [51]

    Explicit-memory multiresolution adaptive framework for speech and music separation,

    A. Bellur, K. Thakkar, and M. Elhilali, “Explicit-memory multiresolution adaptive framework for speech and music separation,”Eurasip Journal on Audio, Speech, and Music Processing, vol. 2023, 2023

    2023

  44. [52]

    On the effectiveness of enrollment speech augmentation for target speaker extraction,

    J. Li, K. Zhang, S. Wang, H. Li, M.-W. Mak, and K. A. Lee, “On the effectiveness of enrollment speech augmentation for target speaker extraction,” in2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, pp. 325–332

    2024

  45. [53]

    Selective listening by synchronizing speech with lips,

    Z. Pan, R. Tao, C. Xu, and H. Li, “Selective listening by synchronizing speech with lips,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 1650–1664, 2022

    2022

  46. [54]

    Robust Speaker Extraction Network Based on Iterative Refined Adap- tation,

    C. Deng, S. Ma, Y . Sha, Y . Zhang, H. Zhang, H. Song, and F. Wang, “Robust Speaker Extraction Network Based on Iterative Refined Adap- tation,” inProc. Interspeech 2021

    2021

  47. [55]

    Listening and group- ing: an online autoregressive approach for monaural speech separation,

    Z.-X. Li, Y . Song, L.-R. Dai, and I. McLoughlin, “Listening and group- ing: an online autoregressive approach for monaural speech separation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 4, pp. 692–703, 2019

    2019

  48. [56]

    Source-aware context network for single-channel multi-speaker speech separation,

    Z.-X. Li, Y .Song, L-R.Dai, and I.McLoughlin, “Source-aware context network for single-channel multi-speaker speech separation,” in2018 IEEE ICASSP

  49. [57]

    An online speaker-aware speech separation approach based on time-domain representation,

    H. Wang, Y . Song, Z.-X. Li, I. McLoughlin, and L.-R. Dai, “An online speaker-aware speech separation approach based on time-domain representation,” inICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6379–6383

    2020

  50. [58]

    Iterative autoregression: a novel trick to improve your low-latency speech enhancement model,

    P. Andreev, N. Babaev, A. Saginbaev, I. Shchekotov, and A. Alanov, “Iterative autoregression: a novel trick to improve your low-latency speech enhancement model,” inINTERSPEECH 2023, 2023

    2023

  51. [59]

    Neuroheed: Neuro- steered speaker extraction using eeg signals,

    Z. Pan, M. Borsdorf, S. Cai, T. Schultz, and H. Li, “Neuroheed: Neuro- steered speaker extraction using eeg signals,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  52. [60]

    Neuroheed+: Improving neuro-steered speaker extraction with joint auditory attention detection,

    Z. Pan, G. Wichern, F. G. Germain, S. Khurana, and J. Le Roux, “Neuroheed+: Improving neuro-steered speaker extraction with joint auditory attention detection,” in2024 IEEE ICASSP

  53. [61]

    Paris: Pseudo-autoregressive siamese training for online speech separation,

    Z. Pan, G. Wichern, F. G. Germain, K. Saijo, and J. Le Roux, “Paris: Pseudo-autoregressive siamese training for online speech separation,” in Interspeech 2024

    2024

  54. [62]

    On- line Audio-Visual Autoregressive Speaker Extraction,

    Z. Pan, W. Wang, S. Zhao, C. Zhang, K. Zhou, Y . Ma, and B. Ma, “On- line Audio-Visual Autoregressive Speaker Extraction,” inInterspeech 2025, 2025, pp. 1928–1932

    2025

  55. [63]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  56. [64]

    Ecapa-tdnn: Em- phasized channel attention, propagation and aggregation in tdnn based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa-tdnn: Em- phasized channel attention, propagation and aggregation in tdnn based speaker verification,” inInterspeech 2020, 2020, pp. 3830–3834

    2020

  57. [65]

    Wespeaker: A research and production oriented speaker embedding learning toolkit,

    H. Wang, C. Liang, S. Wang, Z. Chen, B. Zhang, X. Xiang, Y . Deng, and Y . Qian, “Wespeaker: A research and production oriented speaker embedding learning toolkit,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

    2023

  58. [66]

    Curriculum learning,

    Y . Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” inProceedings of the 26th annual international conference on machine learning, 2009, pp. 41–48

    2009

  59. [67]

    A learning algorithm for continually running fully recurrent neural networks,

    R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,”Neural computation, vol. 1, no. 2, pp. 270–280, 1989

    1989

  60. [68]

    Scheduled sampling for sequence prediction with recurrent neural networks,

    S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling for sequence prediction with recurrent neural networks,”Advances in neural information processing systems, vol. 28, 2015

    2015

  61. [69]

    Delving into high-quality syn- thetic face occlusion segmentation datasets,

    K. T. R. V oo, L. Jiang, and C. C. Loy, “Delving into high-quality syn- thetic face occlusion segmentation datasets,”2022 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4710–4719, 2022

    2022

  62. [70]

    Watch or listen: Robust audio-visual speech recognition with visual corruption modeling and reliability scoring,

    J. Hong, M. Kim, J. Y . Choi, and Y . M. Ro, “Watch or listen: Robust audio-visual speech recognition with visual corruption modeling and reliability scoring,”2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  63. [71]

    V oxceleb2: Deep speaker recognition,

    J. S. Chung, A. Nagrani, and A. Zisserman, “V oxceleb2: Deep speaker recognition,” inINTERSPEECH, 2018

    2018

  64. [72]

    Avhumar: Audio-visual target speech extraction with pre-trained av-hubert and mask-and-recover strategy,

    W. Wu, X. Chen, X. Wu, H. Li, and H. Meng, “Avhumar: Audio-visual target speech extraction with pre-trained av-hubert and mask-and-recover strategy,”CVPR 2024 Sight and Sound Workshop, 2024

    2024

  65. [73]

    Target speech extraction with pre-trained av-hubert and mask-and-recover strategy,

    W. Wu, X. Chen, X. Wu, H. Li, and H. Meng, “Target speech extraction with pre-trained av-hubert and mask-and-recover strategy,” in2024 International Joint Conference on Neural Networks (IJCNN), 2024

    2024

  66. [74]

    Sdr–half-baked or well done?

    J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “Sdr–half-baked or well done?” in2019 IEEE ICASSP

  67. [75]

    Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,

    A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,” in2001 IEEE ICASSP

  68. [76]

    A short- time objective intelligibility measure for time-frequency weighted noisy speech,

    C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “A short- time objective intelligibility measure for time-frequency weighted noisy speech,” in2010 IEEE ICASSP

  69. [77]

    Time domain audio visual speech separation,

    J. Wu, Y . Xu, S.-X. Zhang, L.-W. Chen, M. Yu, L. Xie, and D. Yu, “Time domain audio visual speech separation,” in2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019, pp. 667–673

    2019

  70. [78]

    Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation,

    Y . Luo and N. Mesgarani, “Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 8, pp. 1256–1266, 2019

    2019

  71. [79]

    Music source separation with band-split rnn,

    Y . Luo and J. Yu, “Music source separation with band-split rnn,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 1893–1901, 2023

    1901

  72. [80]

    Wesep: A scalable and flexible toolkit towards generalizable target speaker extraction,

    S. Wang, K. Zhang, S. Lin, J. Li, X. Wang, M. Ge, J. Yu, Y . Qian, and H. Li, “Wesep: A scalable and flexible toolkit towards generalizable target speaker extraction,” inProc. Interspeech 2024, 2024, pp. 4273– 4277

    2024

  73. [81]

    Audio-visual target speaker extraction with selective auditory attention,

    R. Tao, X. Qian, Y . Jiang, J. Li, J. Wang, and H. Li, “Audio-visual target speaker extraction with selective auditory attention,”IEEE Transactions on Audio, Speech and Language Processing, vol. 33, pp. 797–811, 2025

    2025

  74. [82]

    Attention is all you need in speech separation,

    C. Subakan, M. Ravanelli, S. Cornell, M. Bronzi, and J. Zhong, “Attention is all you need in speech separation,” inICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 21–25

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.