RECORD
Towards Robust Real-time Audio-Visual Speech Enhancement
Sound · Submitted 16 Dec 2021
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Comparative simulation results show that our real time AV SE framework outperforms state-of-the-art SE approaches, including recent DNN based SE models.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
The human brain contextually exploits heterogeneous sensory information to efficiently perform cognitive tasks including vision and hearing. For example, during the cocktail party situation, the human auditory cortex contextually integrates audio-visual (AV) cues in order to better perceive speech. Recent studies have shown that AV speech enhancement (SE) models can significantly improve speech quality and intelligibility in very low signal to noise ratio (SNR) environments as compared to audio-only SE models. However, despite significant research in the area of AV SE, development of real-time processing models with low latency remains a formidable technical challenge. In this paper, we present a novel framework for low latency speaker-independent AV SE that can generalise on a range of visual and acoustic noises. In particular, a generative adversarial networks (GAN) is proposed to address the practical issue of visual imperfections in AV SE. In addition, we propose a deep neural network based real-time AV SE model that takes into account the cleaned visual speech output from GAN to deliver more robust SE. The proposed framework is evaluated on synthetic and real noisy AV corpora using objective speech quality and intelligibility metrics and subjective listing tests. Comparative simulation results show that our real time AV SE framework outperforms state-of-the-art SE approaches, including recent DNN based SE models.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | The human brain contextually exploits heterogeneous sensory information to efficiently perform cognitive tasks including vision and hearing. | background 1.00 | 0.03 | 0.01 | asserted 1.00 |
| s1 | For example, during the cocktail party situation, the human auditory cortex contextually integrates audio-visual (AV) cues in order to better perceive speech. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s2 | Recent studies have shown that AV speech enhancement (SE) models can significantly improve speech quality and intelligibility in very low signal to noise ratio (SNR) environments as compared to audio-only SE models. | background 1.00 | 0.04 | 0.05 | asserted 1.00 |
| s3 | However, despite significant research in the area of AV SE, development of real-time processing models with low latency remains a formidable technical challenge. | gap 0.93 | 0.05 | 0.03 | asserted 1.00 |
| s4 | In this paper, we present a novel framework for low latency speaker-independent AV SE that can generalise on a range of visual and acoustic noises. | contribution 1.00 | 0.87 | 0.12 | asserted 1.00 |
| s5 | In particular, a generative adversarial networks (GAN) is proposed to address the practical issue of visual imperfections in AV SE. | method 0.76 | 0.85 | 0.03 | asserted 0.97 |
| s6 | In addition, we propose a deep neural network based real-time AV SE model that takes into account the cleaned visual speech output from GAN to deliver more robust SE. | method 0.81 | 0.92 | 0.06 | asserted 0.98 |
| s7 | The proposed framework is evaluated on synthetic and real noisy AV corpora using objective speech quality and intelligibility metrics and subjective listing tests. | method 0.98 | 0.95 | 0.04 | asserted 0.93 |
| s8 | Comparative simulation results show that our real time AV SE framework outperforms state-of-the-art SE approaches, including recent DNN based SE models. | result 1.00 | 0.95 | 0.85 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
The human brain contextually exploits heterogeneous sensory information to efficiently perform cognitive tasks including vision and hearing. For example, during the cocktail party situation, the human auditory cortex contextually integrates audio-visual (AV) cues in order to better perceive speech. Recent studies have shown that AV speech enhancement (SE) models can significantly improve speech quality and intelligibility in very low signal to noise ratio (SNR) environments as compared to audio-only SE models. However, despite significant research in the area of AV SE, development of real-time processing models with low latency remains a formidable technical challenge. In this paper, we present a novel framework for low latency speaker-independent AV SE that can generalise on a range of visual and acoustic noises. In particular, a generative adversarial networks (GAN) is proposed to address the practical issue of visual imperfections in AV SE. In addition, we propose a deep neural network based real-time AV SE model that takes into account the cleaned visual speech output from GAN to deliver more robust SE. The proposed framework is evaluated on synthetic and real noisy AV corpora using objective speech quality and intelligibility metrics and subjective listing tests. Comparative simulation results show that our real time AV SE framework outperforms state-of-the-art SE approaches, including recent DNN based SE models.
Human review
How the literature uses it
2 arXiv papers cite this in 2 sentences
Papers citing it · 1 sentence without a confident label
Cites it as background 1 paper
-
Specifically, in the domain of audiovisual learning, recent years have seen determined efforts to build robust models for tasks including but not limited to: speech recognition [ref] , expression recognition [ref] , event localization [ref] , voice activity detection [ref] , video classification [ref] , and speech enhancement [ref] .
On Robustness to Missing Video for Audiovisual Speech Recognition · Introduction · 0.86
What it cites · 4 sentences without a confident label
Uses the method or tool of 1 paper
-
The 3D-CNN features are fed to standard RESNET-18 architecture [ref] .
Resnet in Resnet: Generalizing Residual Architectures · Lip Embedding Network · 1.00
Uses data or values from 1 paper
-
Each of the utterances are mixed randomly with the noises from MUSAN [ref] .
MUSAN: A Music, Speech, and Noise Corpus · Synthetic Dataset · 0.98
Cites as background 5 papers
-
In the literature, extensive studies have been carried out to develop AV SE methods in time-domain and frequency domain [ref] .
The Conversation: Deep Audio-Visual Speech Enhancement · Introduction · 0.99
-
In the literature, extensive studies have been carried out to develop AV SE methods in time-domain and frequency domain [ref] .
My lips are concealed: Audio-visual speech enhancement through obstructions · Introduction · 0.99
-
In the literature, extensive studies have been carried out to develop AV SE methods in time-domain and frequency domain [ref] .
On the Role of Visual Cues in Audiovisual Speech Enhancement · Introduction · 0.99
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.