Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RAVEN, a real-time audio-visual speech enhancement system, shows that the optimal pre-trained visual front-end depends on the acoustic condition: AVSR plus ASD embeddings win in low-SNR multi-speaker mixtures, while AVSR embeddings alone…

desk verdict A useful open-source real-time AVSE release, but the embedding-ranking claims need error bars before they are convincing. read the letter →

arxiv 2507.21448 v2 pith:J7RRMFVY submitted 2025-07-29 eess.AS cs.ETcs.LG

T. Aleksandra Ma , Sile Yin , Li-Chia Yang , Shuo Zhang This is my paper
classification eess.AScs.ETcs.LG
keywords audio-visualspeechenhancementreal-timepre-trainedvisualembeddingsrecognitionactivespeakerdetectionmask-basedfusionphase-sensitivelossVoxCeleb2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RAVEN is a real-time audio-visual speech enhancement system that reuses frozen visual embeddings from models trained for other speech tasks rather than learning video features from scratch. The paper's central claim is that the best visual front-end depends on the acoustic scenario: in noise-only mixtures the VSRiW audio-visual speech recognition embedding performs best, while in low-SNR mixtures with one or three interfering speakers the concatenation of AV-HuBERT (speech recognition) and TalkNet (speaker detection) embeddings gives the largest gains. The authors also claim their implementation is the first open-source AVSE system that runs in real time on a CPU, with per-frame processing under 40 milliseconds. If right, this gives practitioners a direct recipe for choosing pre-trained visual features in deployed AVSE, plus a reproducible baseline for future work.

What carries the argument

The core mechanism is mask-based late fusion. An audio stream converts the 16 kHz mixture into a compressed complex spectrogram via STFT and a five-layer CNN; a video stream extracts embeddings from a frozen pre-trained visual encoder (AVSR or ASD) at 25 fps, upsampled to audio-frame rate. The two feature sets are concatenated and passed through a unidirectional LSTM and three fully connected layers that predict a magnitude mask, which is combined with the mixture phase to reconstruct the complex spectrogram. Training uses a phase-sensitive loss (MSE on both magnitude and compressed complex spectrogram). The real-time system buffers five video frames, giving a two-frame lookahead and 120 ms algorithmic latency, while each frame's processing stays under the 40 ms streaming interval.

What would settle it

Re-run the training and evaluation on a dataset with genuine clean references (for example, LRS3's original audio or studio-recorded corpora) and check whether the ranking of visual front-ends — VSRiW best in noise-only, AV-HuBERT plus TalkNet best in low-SNR multi-speaker — persists and whether the absolute gains remain when ground truth is not produced by DeepFilterNet.

Watch

Extended reading notes

Core claim

The central discovery is that visual embeddings trained for audio-visual speech recognition (AVSR) and active speaker detection (ASD) transfer to speech enhancement, and that their relative value is condition-dependent. In noise-only conditions, VSRiW (an AVSR model) achieves the best PESQ, SISDR, and ESTOI among all tested front-ends. In low-SNR, multi-speaker conditions, concatenating AV-HuBERT (AVSR) and TalkNet (ASD) embeddings outperforms either alone, with the largest relative gains in SISDR and ESTOI. The authors attribute the benefit to complementary information: continuous lip-movement features from AVSR plus target-speaker activity cues from ASD. They also report that the model generalizes to multilingual in-the-wild data but degrades on emotionally expressive speech, and that the system meets real-time constraints on a CPU with an algorithmic latency of 120 ms caused by the two-frame video lookahead.

Load-bearing premise

The training targets and all evaluation scores are measured against DeepFilterNet-denoised VoxCeleb2 audio, not against true clean studio recordings; the paper notes this produces 'reasonably clean speech for most utterances upon inspection' but does not establish that this reference is faithful to the original clean speech.

Editorial extensions

If this is right

  • A pragmatic rule for deployment: use VSRiW-style AVSR embeddings in noise-only conditions and AV-HuBERT plus TalkNet embeddings in low-SNR, multi-speaker conditions.
  • Real-time AVSE is achievable on a CPU with frozen pre-trained visual features, so the system can be reproduced and integrated without specialized hardware.
  • Cross-task concatenation (AVSR + ASD) helps more in multi-speaker mixtures than same-task concatenation, suggesting that activity-level cues and phonetic lip-motion cues are complementary.
  • The released code provides a public baseline for real-time AVSE, allowing future work to benchmark against a common implementation.
  • Generalization tests indicate the approach transfers across languages but not across strong emotional expressions, marking emotion robustness as the next obstacle.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: replace AV-HuBERT with an audio-only HuBERT or a differently noise-augmented AVSR model to determine whether its multi-speaker advantage comes from the AVSR objective or from its diverse noise augmentation during training.
  • The condition-dependent ranking suggests a deployment strategy: a system that switches front-ends based on estimated SNR and number of active speakers could get the best of both regimes.
  • Because all metrics are computed against DeepFilterNet-denoised audio, re-scoring on studio-clean targets would show how much of the reported ranking is tied to the denoiser's definition of clean.
  • The 120 ms latency is dominated by video lookahead; a system that trades accuracy for latency by reducing the lookahead could open the design to more interactive applications such as live captioning or hearing aids.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents RAVEN, a real-time audio-visual speech enhancement (AVSE) system that combines a causal audio stream with pre-trained visual embeddings from audio-visual speech recognition (AVSR) and active speaker detection (ASD) models. Four visual front-ends are evaluated (AV-HuBERT, VSRiW, TalkNet, LoCoNet) along with concatenations of the best from each task. The central empirical claim is that concatenating AVSR and ASD embeddings (AV-HuBERT+TalkNet) yields the greatest improvement in low-SNR, multi-speaker conditions, while AVSR embeddings alone (VSRiW) perform best in noise-only scenarios. The system reportedly operates on a CPU with a processing time of 28 ms per frame, and the code is released. The paper also reports generalization results on ViSPeR and MEAD datasets.

Significance. If the comparative results are robust, the paper would make a useful contribution by identifying which pre-trained visual features are most suitable for different acoustic conditions and by providing the first open-source real-time AVSE implementation. The reproducible code, video demonstration, and systematic comparison across visual encoders are explicit strengths. However, the central ranking claim rests on very small PESQ/ESTOI/SISDR differences without statistical validation, and the evaluation uses DeepFilterNet-denoised audio as ground truth, which threatens the validity of absolute and relative quality measurements. The significance is therefore conditional on addressing these methodological concerns.

major comments (4)
  1. [Section 4.1, Tables 1 and 2] The central claim that concatenating AVSR and ASD embeddings is 'best' in low-SNR, multi-speaker settings is not statistically supported. In Table 2, at -10 dB with three interfering speakers, AVH+T has PESQ 1.096 versus AVHuBERT 1.097 (AVHuBERT is actually higher), and at -10 dB with one speaker the PESQ difference between AVH+T and AVH+V is 0.000. ESTOI differences range from 0.001 to 0.008. No error bars, confidence intervals, or significance tests are reported, and all models are trained for only 5 epochs, so run-to-run stochastic variation can easily exceed these margins. The paper should either provide repeated-seed results with paired significance tests or explicitly state that the observed differences are within noise.
  2. [Section 3.1] The ground-truth clean speech is generated by denoising VoxCeleb2 utterances with DeepFilterNet, and all PESQ, SISDR, and ESTOI scores in Tables 1-3 are computed relative to this model-generated reference. This is a load-bearing methodological choice: if DeepFilterNet introduces artifacts, removes speech components, or behaves differently for certain accents or emotions, the reported improvements may reflect how well the model mimics DeepFilterNet's processing rather than true clean-speech quality. The paper should validate that DeepFilterNet outputs are acceptable clean references (e.g., by comparing against human-rated clean speech or using a separate clean-speech dataset) and discuss the potential bias this introduces.
  3. [Section 3.1 and Table 2] The paper states the input SNR range is -10 to 10 dB, but Table 2 only reports results at -10 dB and -5 dB for multi-speaker conditions. The central claim concerns 'low-SNR, multi-speaker environments,' which is only partially covered by these two points. Without results at 0, 5, or 10 dB, it is unclear whether the advantage of concatenating AVSR and ASD embeddings persists across the low-SNR range or is limited to the most adverse conditions. Please report results at intermediate SNRs or justify the omission.
  4. [Section 4.1] No audio-only baseline is reported. The paper motivates AVSE by the limitations of audio-only enhancement, but the reader cannot assess whether the visual embeddings provide any benefit over the audio stream alone. All evaluated models include both audio and visual inputs, so the contribution of the visual modality is not quantified. Adding an audio-only version of the fusion model (i.e., without the visual stream) would provide a critical reference point and strengthen the claim that the pre-trained visual representations are useful for AVSE.
minor comments (4)
  1. [Table 2] The row label 'AVH+T' and the caption's 'AVHuBERT+TalkNet' are not defined in the table; please define all abbreviations in the caption or a table footnote.
  2. [Section 2.3] The real-time evaluation reports the processing time only for VSRiW (28 ms per frame). Please report processing times for all visual encoders to substantiate the claim that the system 'meets the real-time requirement for all pre-trained embeddings'.
  3. [Section 3.2] The test set size is not reported. Please specify the number of test utterances for VoxCeleb2, ViSPeR, and MEAD, as metric variability depends on sample size.
  4. [Section 2.1] The audio CNN is described as 'not causal' while the system is claimed to be real-time. Please clarify that the non-causality is limited to a two-frame lookahead (40 ms) that is included in the reported algorithmic latency.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: held-out test evaluation with external metrics; only one minor non-load-bearing self-citation.

full rationale

The derivation chain is self-contained and empirically grounded. The model is trained on mixture inputs to predict a fixed target (DeepFilterNet-denoised VoxCeleb2 audio, Sec. 3.1) and is evaluated on held-out test mixtures with external PESQ/SISDR/ESTOI metrics; no fitted parameter from the test set is renamed as a prediction. The four visual front-ends (AV-HuBERT, VSRiW, TalkNet, LoCoNet) are pretrained on AVSR/ASD tasks rather than on AVSE and are used as frozen feature extractors, so the comparison between them is not an identity or a self-fulfilling construction. The headline ranking across noise-only and multi-speaker conditions is a descriptive selection from Tables 1-2 rather than a prospective prediction, which is a statistical-robustness caveat but not circularity. The only self-citation is [29] in Sec. 2.2.3, used as motivation for concatenating pretrained embeddings; the AVSR+ASD concatenation result is established by the paper's own experiments, so the citation is not load-bearing. The DeepFilterNet-generated clean reference is a data-quality limitation and could bias absolute metric values, but it is shared across all compared encoders and does not make any result equivalent to its input by construction. No equation in the paper reduces to its own input, and no prior uniqueness theorem or author-specific result is invoked to force the architectural choice.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three kinds of inputs: standard STFT/mask processing, pre-trained external visual encoders, and a model-generated clean-speech reference. The first two are external and reproducible; the last is an assumption that could bias every reported number. The only hand-set values are standard hyperparameters, not fitted constants.

free parameters (3)
  • Power-law compression rate p = 0.3
    Applied to complex spectrograms before loss and mask estimation; adopted from prior work (Ephrat et al.) and not tuned in this paper.
  • STFT configuration = window 400, hop 160, nfft 512
    Time-frequency analysis settings stated in Section 2.1; standard choices from the speech enhancement literature, not fitted to the data.
  • Training epochs = 5
    All fusion models trained for 5 epochs (Section 4.1); a hand-chosen schedule without reported early stopping or convergence analysis.
assumptions (4)
  • domain assumption DeepFilterNet output approximates true clean speech for VoxCeleb2 utterances.
    Invoked in Section 3.1 to create training targets and evaluation references; only informally verified by inspection.
  • domain assumption Pre-trained AVSR and ASD embeddings transfer to the mouth/face crops of VoxCeleb2, ViSPeR, and MEAD speakers.
    Used in Sections 2.2 and 4.2; relies on MediaPipe alignment and same cropped face regions matching the encoders' training distributions.
  • standard math Mask-based enhancement with mixture phase reconstructs intelligible speech.
    Standard STFT-domain assumption in Section 2.1; the phase-sensitive loss (Eq. 1) partially compensates for phase loss.
  • domain assumption The non-causal audio CNN's receptive field fits within the video lookahead, so the system remains causal at the system level.
    Stated in Section 2.1 and quantified in Section 2.3 (120 ms algorithmic latency); assumes frame-aligned buffering.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations." pith.science (2026). https://pith.science/paper/J7RRMFVY

@misc{pith2026250721448,
  author       = {Pith},
  title        = {Pith review of: Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J7RRMFVY}},
  note         = {Machine review of arXiv:2507.21448}
}
read the original abstract

Speech enhancement in audio-only settings remains challenging, particularly in the presence of interfering speakers. This paper presents a simple yet effective real-time audio-visual speech enhancement (AVSE) system, RAVEN, which isolates and enhances the on-screen target speaker while suppressing interfering speakers and background noise. We investigate how visual embeddings learned from audio-visual speech recognition (AVSR) and active speaker detection (ASD) contribute to AVSE across different SNR conditions and numbers of interfering speakers. Our results show concatenating embeddings from AVSR and ASD models provides the greatest improvement in low-SNR, multi-speaker environments, while AVSR embeddings alone perform best in noise-only scenarios. In addition, we develop a real-time streaming system that operates on a computer CPU and we provide a video demonstration and code repository. To our knowledge, this is the first open-source implementation of a real-time AVSE system.

Figures

Figures reproduced from arXiv: 2507.21448 by the authors.

Figure 1
Figure 1. Architecture of our mask-based late fusion method. Each CNN layer in the audio stream is followed by a batch normalization layer and a ReLU layer. The temporal receptive field of the CNN layers is 5, with a 2-frame lookahead. Each fully connected layer is followed by a ReLU layer. A Sigmoid function is applied to the magnitude mask to limit the mask values from 0 to 1. The video stream utilizes the front-end output … view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FSD50K-Solo: Automated Curation of Single-Source Sound Events

    eess.AS 2026-05 conditional novelty 6.0 of 10

    The authors present a scalable curation method that combines diffusion-based mixture synthesis with a discriminative classifier to automatically extract single-source sound events from FSD50K and release the cleaned F...

  2. FSD50K-Solo: Automated Curation of Single-Source Sound Events

    eess.AS 2026-05 unverdicted novelty 6.0 of 10

    A curation pipeline combining diffusion-based synthetic mixtures with a discriminative classifier produces and releases FSD50K-Solo, a single-source subset of FSD50K that matches human expert labels on a test set.

Reference graph

Works this paper leans on

41 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    Introduction Speech enhancement is essential in various applications, both as a standalone task and as a component of broader speech- related systems. It can improve speech quality in phone calls and hearing devices, as well as enhance downstream tasks such as automatic speech recognition (ASR) and speech-based bio- metric authentication. In academic rese...

    work page

  2. [2]

    Audio-visual fusion model As shown in Fig

    Methodology 2.1. Audio-visual fusion model As shown in Fig. 1, our proposed model is a mask-based late fusion approach [19] that incorporates phase information to im- prove speech enhancement. The audio stream transforms the raw 16 kHz input into the time-frequency domain using the Short-Time Fourier Transform (STFT). The resulting spectro- grams are pass...

    arXiv 2025

  3. [3]

    Datasets We use V oxCeleb2 as the speech dataset and MUSAN for noise and music

    Experiment setup 3.1. Datasets We use V oxCeleb2 as the speech dataset and MUSAN for noise and music. V oxCeleb2 is an audio-visual speech dataset containing approximately one million utterances from vari- ous celebrities, extracted from videos uploaded to YouTube recorded in diverse acoustic environments. The dataset is mul- tilingual and includes uttera...

    work page

  4. [4]

    Results and discussion 4.1. Visual encoder performance analysis We present our results as follows: Table 1 presents the perfor- mance of our models with different visual encoders in a noise- only scenario with SNR uniformly sampled from [−10 dB, 10 dB]. Table 2 reports performance in low-SNR, multi-speaker settings. All fusion models were trained for 5 ep...

    work page

  5. [5]

    We systemat- ically analyze how visual embeddings from audio-visual speech recognition (A VSR) and active speaker detection (ASD) im- pact A VSE performance

    Conclusion In this work, we propose a simple yet effective phase-aware audiovisual fusion network that leverages visual embeddings from other speech tasks for speech enhancement. We systemat- ically analyze how visual embeddings from audio-visual speech recognition (A VSR) and active speaker detection (ASD) im- pact A VSE performance. Our results indicate...

    work page

  6. [6]

    Funda- mentals, present and future perspectives of speech enhancement,

    N. Das, S. Chakraborty, J. Chaki, N. Padhy, and N. Dey, “Funda- mentals, present and future perspectives of speech enhancement,” International Journal of Speech Technology , vol. 24, no. 4, pp. 883–901, 2021

    work page 2021

  7. [7]

    An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation,

    D. Michelsanti, Z.-H. Tan, S.-X. Zhang, Y . Xu, M. Yu, D. Yu, and J. Jensen, “An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, pp. 1368– 1396, 2021

    work page 2021

  8. [8]

    FlowA VSE: Effi- cient Audio-Visual Speech Enhancement with Conditional Flow Matching,

    C. Jung, S. Lee, J.-H. Kim, and J. S. Chung, “FlowA VSE: Effi- cient Audio-Visual Speech Enhancement with Conditional Flow Matching,” in Interspeech, 2024

    work page 2024

Show all 41 references
  1. [9]

    Personalized speech enhancement: new models and Comprehensive evaluation,

    S. E. Eskimez, T. Yoshioka, H. Wang, X. Wang, Z. Chen, and X. Huang, “Personalized speech enhancement: new models and Comprehensive evaluation,” in International Conference on Acoustics, Speech & Signal Processing (ICASSP), May 2022

    2022

  2. [10]

    Real-Time Audio-Visual End-to-End Speech Enhancement,

    Z. Zhu, H. Yang, M. Tang, Z. Yang, S. E. Eskimez, and H. Wang, “Real-Time Audio-Visual End-to-End Speech Enhancement,” in International Conference on Acoustics, Speech & Signal Process- ing (ICASSP), 2023

    2023

  3. [11]

    Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation,

    A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Has- sidim, W. T. Freeman, and M. Rubinstein, “Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation,” ACM Transactions on Graphics, vol. 37, no. 4, pp. 1–11, 2018

    2018

  4. [12]

    The Conversation: Deep Audio-Visual Speech Enhancement,

    T. Afouras, J. S. Chung, and A. Zisserman, “The Conversation: Deep Audio-Visual Speech Enhancement,” inInterspeech, 2018

    2018

  5. [13]

    Improved Lite Audio- Visual Speech Enhancement,

    S.-Y . Chuang, H.-M. Wang, and Y . Tsao, “Improved Lite Audio- Visual Speech Enhancement,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022

    2022

  6. [14]

    Lite Audio- Visual Speech Enhancement,

    S.-Y . Chuang, Y . Tsao, C.-C. Lo, and H.-M. Wang, “Lite Audio- Visual Speech Enhancement,” inInterspeech, 2020

    2020

  7. [15]

    End-to-end Audio-visual Speech Recognition with Conformers,

    P. Ma, S. Petridis, and M. Pantic, “End-to-end Audio-visual Speech Recognition with Conformers,” in International Confer- ence on Acoustics, Speech & Signal Processing (ICASSP), 2021

    2021

  8. [16]

    Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis,

    K. Yang, D. Markovic, S. Krenn, V . Agrawal, and A. Richard, “Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis,” in Conference on Computer Vi- sion and Pattern Recognition (CVPR). New Orleans, LA, USA: IEEE, 2022

    2022

  9. [17]

    A Novel Real-Time, Lightweight Chaotic-Encryption Scheme for Next- Generation Audio-Visual Hearing Aids,

    A. Adeel, J. Ahmad, H. Larijani, and A. Hussain, “A Novel Real-Time, Lightweight Chaotic-Encryption Scheme for Next- Generation Audio-Visual Hearing Aids,”Cognitive Computation, vol. 12, no. 3, pp. 589–601, 2020

    2020

  10. [18]

    Lip- Reading Driven Deep Learning Approach for Speech Enhance- ment,

    A. Adeel, M. Gogate, A. Hussain, and W. M. Whitmer, “Lip- Reading Driven Deep Learning Approach for Speech Enhance- ment,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 5, no. 3, pp. 481–490, Jun. 2021

    2021

  11. [19]

    Audio-visual speech enhancement using deep neural networks,

    J.-C. Hou, S.-S. Wang, Y .-H. Lai, J.-C. Lin, Y . Tsao, H.-W. Chang, and H.-M. Wang, “Audio-visual speech enhancement using deep neural networks,” inAsia-Pacific Signal and Information Process- ing Association Annual Summit and Conference (APSIPA), 2016

    2016

  12. [20]

    Audio-Visual Scene Analysis with Self-Supervised Multisensory Features,

    A. Owens and A. A. Efros, “Audio-Visual Scene Analysis with Self-Supervised Multisensory Features,” in European Conference on Computer Vision, V . Ferrari, M. Hebert, C. Sminchisescu, and Y . Weiss, Eds., 2018

    2018

  13. [21]

    Evaluating Audiovisual Source Separation in the Context of Video Conferencing,

    B. ˙Inan, M. Cernak, H. Grabner, H. P. Tukuljac, R. C. Pena, and B. Ricaud, “Evaluating Audiovisual Source Separation in the Context of Video Conferencing,” inInterspeech, 2019

    2019

  14. [22]

    CochleaNet: A robust language-independent audio-visual model for real-time speech enhancement,

    M. Gogate, K. Dashtipour, A. Adeel, and A. Hussain, “CochleaNet: A robust language-independent audio-visual model for real-time speech enhancement,” Information Fusion, vol. 63, pp. 273–285, 2020

    2020

  15. [23]

    RT-LA-V ocE: Real- Time Low-SNR Audio-Visual Speech Enhancement,

    H. Chen, R. Mira, S. Petridis, and M. Pantic, “RT-LA-V ocE: Real- Time Low-SNR Audio-Visual Speech Enhancement,” in Inter- speech, 2024

    2024

  16. [24]

    Exploring Tradeoffs in Models for Low-Latency Speech Enhancement,

    K. Wilson, M. Chinen, J. Thorpe, B. Patton, J. Hershey, R. A. Saurous, J. Skoglund, and R. F. Lyon, “Exploring Tradeoffs in Models for Low-Latency Speech Enhancement,” in International Workshop on Acoustic Signal Enhancement (IWAENC), 2018

    2018

  17. [25]

    Phase- sensitive and recognition-boosted speech separation using deep recurrent neural networks,

    H. Erdogan, J. R. Hershey, S. Watanabe, and J. Le Roux, “Phase- sensitive and recognition-boosted speech separation using deep recurrent neural networks,” inInternational Conference on Acous- tics, Speech & Signal Processing (ICASSP), 2015

    2015

  18. [26]

    On The Compen- sation Between Magnitude and Phase in Speech Separation,

    Z.-Q. Wang, G. Wichern, and J. L. Roux, “On The Compen- sation Between Magnitude and Phase in Speech Separation,” IEEE Signal Processing Letters , vol. 28, pp. 2018–2022, 2021, arXiv:2108.05470 [cs]

    2018 arXiv

  19. [27]

    Learn- ing Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction,

    B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learn- ing Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction,” inInternational Conference on Learning Rep- resentations (ICLR), 2022

    2022

  20. [28]

    Robust Self-Supervised Audio-Visual Speech Recognition,

    B. Shi, W.-N. Hsu, and A. Mohamed, “Robust Self-Supervised Audio-Visual Speech Recognition,” inInterspeech, 2022

    2022

  21. [29]

    Visual Speech Recognition for Multiple Languages in the Wild,

    P. Ma, S. Petridis, and M. Pantic, “Visual Speech Recognition for Multiple Languages in the Wild,” Nature Machine Intelligence , vol. 4, no. 11, pp. 930–939, 2022, arXiv:2202.13084 [cs]

    2022 arXiv

  22. [30]

    Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection,

    R. Tao, Z. Pan, R. K. Das, X. Qian, M. Z. Shou, and H. Li, “Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection,” in ACM International Conference on Multimedia, 2021, arXiv:2107.06592 [eess]

    2021 arXiv

  23. [31]

    LoCoNet: Long-Short Context Network for Active Speaker Detection,

    X. Wang, F. Cheng, G. Bertasius, and D. Crandall, “LoCoNet: Long-Short Context Network for Active Speaker Detection,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  24. [32]

    Deep Contextualized Word Representa- tions,

    M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep Contextualized Word Representa- tions,” in Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long Papers...

    2018

  25. [33]

    Few- shot Image Classification: Just Use a Library of Pre-trained Fea- ture Extractors and a Simple Classifier,

    A. Chowdhury, M. Jiang, S. Chaudhuri, and C. Jermaine, “Few- shot Image Classification: Just Use a Library of Pre-trained Fea- ture Extractors and a Simple Classifier,” in International Confer- ence on Computer Vision (ICCV), 2021

    2021

  26. [34]

    Music auto-tagging in the long tail: A few-shot approach,

    T. A. Ma and A. Lerch, “Music auto-tagging in the long tail: A few-shot approach,” in Proceedings of the Audio Engineering So- ciety NY Show, 2024

    2024

  27. [35]

    Contextual String Em- beddings for Sequence Labeling,

    A. Akbik, D. Blythe, and R. V ollgraf, “Contextual String Em- beddings for Sequence Labeling,” in International Conference on Computational Linguistics, 2018

    2018

  28. [36]

    DeepFilterNet: A Low Complexity Speech Enhancement Frame- work for Full-Band Audio based on Deep Filtering,

    H. Schr ¨oter, A. N. Escalante-B, T. Rosenkranz, and A. Maier, “DeepFilterNet: A Low Complexity Speech Enhancement Frame- work for Full-Band Audio based on Deep Filtering,” in Inter- national Conference on Acoustics, Speech & Signal Processing (ICASSP), 2022

    2022

  29. [37]

    Perceptual eval- uation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,

    A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Perceptual eval- uation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” in International Conference on Acoustics, Speech & Signal Processing (ICASSP), vol. 2, 2001

    2001

  30. [38]

    SDR – Half-baked or Well Done?

    J. L. Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR – Half-baked or Well Done?” in International Conference on Acoustics, Speech & Signal Processing (ICASSP), 2019

    2019

  31. [39]

    An Algorithm for Predicting the In- telligibility of Speech Masked by Modulated Noise Maskers,

    J. Jensen and C. H. Taal, “An Algorithm for Predicting the In- telligibility of Speech Masked by Modulated Noise Maskers,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 24, no. 11, pp. 2009–2022, 2016

    2009

  32. [40]

    ViSpeR: Multilingual Audio-Visual Speech Recognition,

    S. Narayan, Y . A. D. Djilali, A. Singh, E. L. Bihan, and H. Hacid, “ViSpeR: Multilingual Audio-Visual Speech Recognition,” 2024

    2024

  33. [41]

    MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation,

    K. Wang, Q. Wu, L. Song, Z. Yang, W. Wu, C. Qian, R. He, Y . Qiao, and C. C. Loy, “MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation,” in European Conference on Computer Vision, A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds., 2020

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.