Pith. sign in

REVIEW 3 major objections 5 minor 69 references

Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoders

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Conditioning a variational auto-encoder's speech model on lip-region video improves unsupervised speech enhancement by over 2 dB SDR and 0.3 PESQ, beating a supervised method, especially at low SNR.

desk verdict First unsupervised audio-visual generative speech-enhancement model with real gains over audio-only VAE; the headline comparison to the supervised baseline is weakened by a training-noise mismatch. read the letter →

arxiv 1908.02590 v3 pith:DHDPQZLR submitted 2019-08-07 cs.SD cs.LGeess.AS

Mostafa Sadeghi , Simon Leglaive , Xavier Alameda-PIneda , Laurent Girin , Radu Horaud This is my paper · ORCID
classification cs.SDcs.LGeess.AS
keywords audio-visualspeechenhancementconditionalvariationalauto-encoderunsupervisednonnegativematrixfactorizationMonteCarloexpectation-maximizationvisualpriorlip-regionfeatures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that visual speech information from the lip region can be folded into a deep generative model of clean speech to make single-channel speech enhancement work better, without ever training on noisy audio. Using a conditional variational auto-encoder, the authors model the clean speech spectrum as generated from a latent code whose prior and decoder both take the lip video frame as an extra conditioning input. At test time this learned audio-visual prior is combined with a nonnegative matrix factorization noise model, and the clean speech is recovered through a Monte Carlo expectation-maximization algorithm that acts as a probabilistic Wiener filter. On the NTCD-TIMIT and GRID audio-visual corpora, the method is reported to beat the audio-only VAE by more than 2 dB SDR and 0.3 PESQ and to outperform a supervised audio-visual deep network, with the largest gains at the most negative SNRs. The practical stake is that visual conditioning may offer a noise-type-agnostic way to enhance speech in very noisy conditions without collecting matched noisy/clean training pairs.

What carries the argument

The load-bearing object is the audio-visual conditional variational auto-encoder (AV-CVAE), in which the same visual feature vector $v_n$ (a lip-region embedding) conditions both the prior over the latent variable $p(z_n|v_n;\gamma)$ and the generative decoder $p(s_n|z_n,v_n;\theta)$. The training objective is a modified ELBO that adds a second reconstruction term in which the latent code is drawn from the visual prior rather than the approximate posterior, with a trade-off parameter $\alpha$; this shrinks the train/test gap because at test time only the prior network generates $z$. Enhancement couples this speech model with a nonnegative matrix factorization noise model $b_{fn}\sim\mathcal{N}_c(0,(W_b H_b)_{fn})$, a per-frame gain $g_n$, and a Monte Carlo expectation-maximization algorithm using Metropolis-Hastings sampling for the intractable posterior; reconstruction is a posterior-mean probabilistic Wiener filter.

What would settle it

Compare AV-CVAE enhancement on the NTCD-TIMIT test set with pristine videos versus videos whose lip ROI is heavily blurred or time-shifted by two frames. If the reported 2+ dB SDR advantage over the audio-only VAE persists under visual corruption, the visual prior is not the source of the gain; if the advantage vanishes, the claim is corroborated.

Watch

Extended reading notes

Core claim

The central claim is that a generative speech prior learned jointly from clean audio and lip movements — formalized as an audio-visual conditional variational auto-encoder — yields better enhancement than the equivalent audio-only generative prior, and can even outperform a supervised audio-visual mask-prediction network. The generative model writes each speech STFT coefficient as $s_{fn}\mid z_n, v_n \sim \mathcal{N}_c(0, \sigma_f(z_n,v_n))$ with a latent prior $z_{ln}\mid v_n \sim \mathcal{N}(\bar\mu_l(v_n), \bar\sigma_l(v_n))$, so the visual input serves as an informative prior on the latent speech code. The paper reports median SDR and PESQ improvements over the audio-only VAE of more than 2 dB and 0.3, respectively, and average SDR gains of more than 2 dB over the supervised comparison method, while noting that the supervised method retains an edge in STOI at low noise levels. The authors attribute the visual advantage to the fact that the lip region is not corrupted by acoustic noise, making it a reliable guide especially when the audio signal is heavily masked.

Load-bearing premise

The method assumes the test-time video is a clean, frontal, unoccluded view of the lips that stays synchronized with the audio, so the visual-conditioned prior and decoder faithfully describe the speech being uttered.

Editorial extensions

If this is right

  • Since training needs only clean audio and synchronized video, the method generalizes to unseen noise types and levels without ever seeing noisy mixtures at training.
  • Visual conditioning matters most at low input SNR; below roughly 0 dB the AV-CVAE consistently beats the audio-only VAE on SDR, PESQ, and STOI.
  • The supervised comparison method can be outperformed in average SDR by more than 2 dB by the unsupervised AV-CVAE, though the supervised method is more computationally efficient at test time.
  • The choice of visual front-end matters: a small fully-connected network trained end-to-end beats a pre-trained 3D-convolution plus ResNet front-end for this enhancement task.
  • Setting the ELBO weight $\alpha$ below 1 (e.g., 0.9) improves scores because it forces the visual prior to generate latent codes that decode into good speech frames.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If visual conditioning indeed supplies a noise-free guide, the same mechanism could be extended to visual features beyond the lip ROI (jaw, cheeks, head pose) and to other generative speech priors, with gains likely to persist wherever acoustic SNR is very low.
  • The reported sensitivity to the visual front-end suggests a testable corollary: on unconstrained, non-frontal, or partly occluded video, the advantage over audio-only enhancement should shrink; measuring that drop would quantify how much of the gain comes from clean visual information versus from the architecture itself.
  • Because the method trains only on clean audio and video, it could be adapted to a lifelong-learning setting where a user's own clean recordings continuously refine the prior, something supervised mask predictors cannot do without re-collecting noisy pairs.
  • A direct stress test would be to feed the AV-CVAE with deliberately corrupted or time-shifted visual features at test time; if the low-SNR advantage degrades under small visual perturbations, that would confirm the prior is doing the work rather than the decoder acting as an audio-only model in disguise.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper develops an audio-visual conditional variational auto-encoder (AV-CVAE) for single-channel, speaker-independent speech enhancement. A clean-speech prior is learned from synchronized clean audio and lip-region visual features, with both the decoder and the latent prior conditioned on the visual stream. At test time, the frozen generative model is combined with an NMF-based noise model, a frame gain, and a Monte Carlo EM procedure that samples latent codes via Metropolis-Hastings; enhancement is performed by a posterior-averaged Wiener filter. Experiments on NTCD-TIMIT and GRID report median SDR, PESQ, and STOI improvements over the audio-only VAE baseline, and the paper claims superiority over a state-of-the-art supervised audio-visual enhancement method.

Significance. If the results hold, the paper makes a useful contribution by showing that a generative, noise-unsupervised speech prior can be visually conditioned and still improve enhancement at low SNR, an advantage that is not accessible to audio-only generative priors. The method is clearly positioned relative to the VAE/NMF speech-enhancement line, the MCEM inference is described in detail, training is repeated over five initializations, and evaluation uses public datasets and three standard metrics. However, two issues affect the strength of the central claims: the supervised comparison is conducted under a training-noise mismatch, and one of the central objective-function equations appears to contain a nontrivial discrepancy. These are fixable, but they need to be addressed before the headline claims can be accepted as stated.

major comments (3)
  1. [Section VII-D, Figs. 8 and 12] The claim that the unsupervised AV-CVAE outperforms the supervised method [15] is weakened by a training/test noise-domain mismatch. Section VII-D states that [15] was trained on noisy mixtures built from the DEMAND dataset, while the NTCD-TIMIT test set uses six different noise types (Living Room, White, Cafe, Car, Babble, Street), and the GRID test mixtures are generated from those same noise types. The paper itself acknowledges that the DEMAND noise types differ from the test noise types. Since supervised enhancement performance depends on training-noise coverage, the comparison handicaps [15] and conflates model quality with domain mismatch. Because this comparative claim appears in the abstract, Section II, and Section VII, it is load-bearing. I recommend either retraining [15] on the same six noise types and SNR levels used for evaluation, or explicitly restricting the claim to a comparison with a supervised method trained on a different noise corpus. The AV-CVAE versus A-VAE comparison in Figs. 6-8 is not affected by this issue.
  2. [Section V, Eq. (13)] The printed KL-divergence term in Eq. (13) does not appear to equal the negative KL divergence between the Gaussian distributions defined in (4) and (8), as required by Eq. (10). As written, the bracketed term contains ln(sigma_tilde/sigma_bar) - ln(sigma_tilde), which simplifies to -ln(sigma_bar), and it omits the variance-ratio contribution sigma_tilde/sigma_bar and the additive constant that appear in the standard Gaussian KL. Since Eq. (13) is described as the optimized training cost, this is important: if the implementation follows the printed formula, the training objective is not the one derived from Eq. (10), and if the implementation uses the correct KL, the equation should be corrected. Please provide the intermediate derivation or clarify the variance parameterization.
  3. [Section VII-E and Fig. 5] The hyperparameter alpha in the modified objective (12) is a free parameter, and the paper does not state whether its value was selected on the validation set. Figure 5 compares alpha = 1 with alpha = 0.9 (base) and alpha = 0.85 (augmented), and the text immediately says 'In the following we thus use the AV-CVAE network with alpha = 0.9.' If Fig. 5 reports test-set metrics, the choice of alpha is made on the test set, and the later comparisons inherit a test-set selection bias. Please state the model-selection protocol for alpha and for the choice between base and augmented visual features, and report the corresponding validation curves if they exist.
minor comments (5)
  1. [Section VI-C1] The prior p(zn|vn;gamma) is defined in Eq. (8), not in Eq. (7) as stated in the text around Eq. (19); the citation should be corrected.
  2. [Section V] There is a typo in the paragraph after Eq. (11): 'stragety' should be 'strategy'.
  3. [Section VII-A] The text says the NTCD-TIMIT dataset provides 'five noise levels' and then says the experiments use 'six noise levels' (-15, -10, -5, 0, 5, 15 dB). The relation between the five provided levels and the two additional levels synthesized with FaNT should be stated more clearly.
  4. [Section VI-C1 and Algorithm 1] The Metropolis-Hastings proposal variance epsilon^2 in Eq. (18) is never given a value in the implementation details of Section VII-E. Since the sampling quality and the runtime depend strongly on this step size, please report the value or the tuning procedure used.
  5. [Section VII-B] The description of the GRID test set does not specify how the lip ROIs and visual features were extracted for this corpus. Clarifying whether the same 67x67 ROI extraction and the same visual embedding networks were applied is necessary for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the AV-CVAE derivation is self-contained and the central claims rest on external benchmarks.

full rationale

The paper derives a conditional variational auto-encoder speech prior p(s|z,v) with prior p(z|v), a variational ELBO, and an MCEM inference algorithm combined with NMF noise modeling and a posterior-mean Wiener filter. This is a genuine generative-model derivation, not a renaming or a self-referential definition. The central improvement claims (AV-CVAE over A-VAE, and AV-CVAE over the supervised baseline [15]) are empirical results measured on public datasets (NTCD-TIMIT and GRID) against the authors' earlier audio-only VAE and an externally implemented supervised method. No equation reduces to its own input, and no fitted parameter is renamed as a prediction. The mild caveats—the alpha hyperparameter in Eq. (12) is tuned on the validation set, and the supervised baseline [15] was trained on DEMAND noise while tested on NTCD-TIMIT/GRID noise types, a mismatch the paper itself acknowledges ('The noise types of the DEMAND dataset are different than the ones that were used to generate noisy-speech instances...')—are evaluation-fairness and generalization concerns, not circularity. The visual-front-end citation [55] and the reliance on the authors' earlier VAE framework [20] do not carry the load of the novelty; the audio-visual conditioning and its benefits are demonstrated by direct comparison. No self-citation chain forces the conclusion. The paper is therefore given a score of 0: no circularity found.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim relies on standard VAE inference, a simple Gaussian mixture model, and a heuristic training objective; no new entities are introduced. The main burden is the assumed reliability of clean, synchronized visual features and the frequency-independent gain model.

free parameters (4)
  • alpha = 0.9 (base), 0.85 (augmented)
    Trade-off hyperparameter in the modified ELBO (Eq. 12); chosen on the validation set to maximize improvement (Fig. 5) and used in all final comparisons.
  • latent dimension L = 32
    Capacity of the latent speech code, fixed across all models; not swept.
  • NMF noise rank K = 10
    Number of noise spectral patterns in the NMF model (Eq. 14); fixed by the authors.
  • visual embedding dimension M = 128
    Output size of the lip ROI embedding networks; fixed by the authors.
assumptions (5)
  • domain assumption The complex STFT speech coefficients are conditionally zero-mean proper Gaussian with variance given by the decoder network (Eq. 7).
    Standard VAE assumption inherited from [19]; a simplification of real speech distributions.
  • domain assumption The visual stream is clean, frontal, and synchronized with audio, so the latent prior p(z|v) is a reliable conditioning signal (Eq. 8).
    The entire AV-fusion benefit relies on noise-free lip features; the paper defers head motion and occlusions to future work in Section VIII.
  • domain assumption The mixture is a frequency-independent gain times clean speech plus additive NMF noise (Eq. 15).
    Assumes a single scalar gain per frame and additive noise independent of speech; inherited from [20] and not validated on frequency-dependent or reverberant scenarios.
  • ad hoc to paper The modified objective (12) with alpha and the extra prior-sampling term is a suitable training criterion.
    The extra term Ep(z|v)[ln p(s|z,v)] is not a standard ELBO; it is an unproved heuristic introduced to reduce the train/test gap of the prior network.
  • domain assumption The pretrained 3D Conv + ResNet front-end (from [55]) produces visual features that are informative for speech enhancement, despite being trained for speech recognition.
    Used in 'augmented' variants; if the features are tuned to ASR rather than enhancement, the transfer may be suboptimal (the paper actually finds base features perform better, which partially contradicts the assumption).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoders." pith.science (2026). https://pith.science/paper/DHDPQZLR

@misc{pith2026190802590,
  author       = {Pith},
  title        = {Pith review of: Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DHDPQZLR}},
  note         = {Machine review of arXiv:1908.02590}
}
read the original abstract

Variational auto-encoders (VAEs) are deep generative latent variable models that can be used for learning the distribution of complex data. VAEs have been successfully used to learn a probabilistic prior over speech signals, which is then used to perform speech enhancement. One advantage of this generative approach is that it does not require pairs of clean and noisy speech signals at training. In this paper, we propose audio-visual variants of VAEs for single-channel and speaker-independent speech enhancement. We develop a conditional VAE (CVAE) where the audio speech generative process is conditioned on visual information of the lip region. At test time, the audio-visual speech generative model is combined with a noise model based on nonnegative matrix factorization, and speech enhancement relies on a Monte Carlo expectation-maximization algorithm. Experiments are conducted with the recently published NTCD-TIMIT dataset as well as the GRID corpus. The results confirm that the proposed audio-visual CVAE effectively fuses audio and visual information, and it improves the speech enhancement performance compared with the audio-only VAE model, especially when the speech signal is highly corrupted by noise. We also show that the proposed unsupervised audio-visual speech enhancement approach outperforms a state-of-the-art supervised deep learning method.

Figures

Figures reproduced from arXiv: 1908.02590 by the authors.

Figure 1
Figure 1. The A-VAE network used for learning a speech prior using audio data. The encoder network (left) takes as input the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The two V-VAE network variants (base and augmented) for learning speech prior from visual features. A lip ROI is [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Pipeline of the proposed AV-CVAE architecture for learning an audio-visual speech prior for speech enhancement. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Each one of these 67×67 images visualizes the learned values of the weights associated with a node of the first fully-connected layer of the base V-VAE architecture. This first layer contains 512 nodes, hence there is an equal number of images like the ones displayed h…
Figure 5
Figure 5. Figure 5: Performance of AV-CVAE for two different values of [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison of A-VAE and of the two [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: Performance comparison of A-VAE, V-VAE, NMF, [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: PESQ results per noise types. VAE and AV-CVAE. However, it outperforms A-VAE in terms of PESQ for high noise levels. To see how the algorithms perform for each different noise type, we have reported the performance scores computed over all test samples, separately for …
Figure 10
Figure 10. Figure 10: SDR results per noise types. efficiency at testing. Also, notice that this method takes advantage of the dynamics of both the audio and visual data, through the presence of convolutional layers, which is not the case of our method that used fully-connected layers. VII…
Figure 12
Figure 12. Figure 12: Performance comparison of A-VAE, NMF, AV [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 68 canonical work pages

  1. [15]

    Visual speech enhancement,

    Aviv Gabbay, Asaph Shamir, and Shmuel Peleg, “Visual speech enhancement,” in Proc. Conference of the International Speech Com- munication Association (INTERSPEECH) , 2018, pp. 1170–1174

    work page 2018

  2. [1]

    Jae Soo Lim, Speech enhancement, Prentice-Hall Englewood Cliffs, NJ, 1983

    work page 1983

  3. [2]

    Jacob Benesty, Shoji Makino, and Jingdong Chen, Speech enhancement, Springer Science & Business Media, 2006

    2006

  4. [3]

    Loizou, Speech enhancement: theory and practice , CRC press, 2007

    Philipos C. Loizou, Speech enhancement: theory and practice , CRC press, 2007

    work page 2007

  5. [4]

    Visual contribution to speech intelligibility in noise,

    William Sumby and Irwin Pollack, “Visual contribution to speech intelligibility in noise,”The Journal of the Acoustical Society of America, vol. 26, no. 2, pp. 212–215, 1954. 12 15 10 5 0 5 15 SNR (dB) 0.4 0.2 0.0 0.2 0.4 Improvement A-VAE NMF AV-CVAE Supervised (a) PESQ 15 10 5 0 5 15 SNR (dB) 4 2 0 2 4 6 8 10Improvement A-VAE NMF AV-CVAE Supervised (b) ...

    work page 1954

  6. [5]

    Auditory-visual perception of speech,

    Norman Erber, “Auditory-visual perception of speech,” Journal of Speech and Hearing Disorders , vol. 40, no. 4, pp. 481–492, 1975

    work page 1975

  7. [6]

    Quantifying the contri- bution of vision to speech perception in noise,

    Alison MacLeod and Quentin Summerfield, “Quantifying the contri- bution of vision to speech perception in noise,” British Journal of Audiology, vol. 21, no. 2, pp. 131–141, 1987

    work page 1987

  8. [7]

    Noisy speech enhancement with filters estimated from the speaker’s lips,

    Laurent Girin, Gang Feng, and Jean-Luc Schwartz, “Noisy speech enhancement with filters estimated from the speaker’s lips,” in Proc. European Conference on Speech Communication and Technology (EU- ROSPEECH), Madrid, Spain, 1995, pp. 1559–1562

    work page 1995

Show all 69 references
  1. [8]

    Audio-visual enhancement of speech in noise,

    Laurent Girin, Jean-Luc Schwartz, and Gang Feng, “Audio-visual enhancement of speech in noise,” The Journal of the Acoustical Society of America, vol. 109, no. 6, pp. 3007–3020, 2001

    2001

  2. [9]

    Learning joint statistical models for audio-visual fusion and segregation,

    John W. Fisher III, Trevor Darrell, William T. Freeman, and Paul A. Viola, “Learning joint statistical models for audio-visual fusion and segregation,” in Proc. Advances in Neural Information Processing Systems (NIPS), 2001, pp. 772–778

    2001

  3. [10]

    Audio- visual speech enhancement with A VCDCN (audio-visual codebook de- pendent cepstral normalization),

    Sabine Deligne, Gerasimos Potamianos, and Chalapathy Neti, “Audio- visual speech enhancement with A VCDCN (audio-visual codebook de- pendent cepstral normalization),” in Proc. IEEE International Workshop on Sensor Array and Multichannel Signal Processing , 2002, pp. 68–71

    2002

  4. [11]

    Noisy audio feature enhancement using audio-visual speech data,

    Roland Goecke, Gerasimos Potamianos, and Chalapathy Neti, “Noisy audio feature enhancement using audio-visual speech data,” in Proc. IEEE International Conference on Acoustics, Speech, and Signal Pro- cessing (ICASSP), 2002, pp. II–2025–2028

    2002

  5. [12]

    Audio-visual sound separation via hidden Markov models,

    John R. Hershey and Michael Casey, “Audio-visual sound separation via hidden Markov models,” in Proc. Advances in Neural Information Processing Systems (NIPS) , 2002, pp. 1173–1180

    2002

  6. [13]

    Twin- HMM-based audio-visual speech enhancement,

    Ahmed Hussen Abdelaziz, Steffen Zeiler, and Dorothea Kolossa, “Twin- HMM-based audio-visual speech enhancement,” in Proc. IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), 2013, pp. 3726–3730

    2013

  7. [14]

    The conversation: Deep audio-visual speech enhancement,

    Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman, “The conversation: Deep audio-visual speech enhancement,” in Proc. Confer- ence of the International Speech Communication Association (INTER- SPEECH), 2018, pp. 3244–3248

    2018

  8. [16]

    Seeing through noise: Speaker separation and enhancement using visually- derived speech,

    Aviv Gabbay, Ariel Ephart, Tavi Halperin, and Shmuel Peleg, “Seeing through noise: Speaker separation and enhancement using visually- derived speech,” in Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , 2018, pp. 3051–3055

    2018

  9. [17]

    Audio-visual speech enhancement using multimodal deep convolutional neural networks,

    Jen-Cheng Hou, Syu-Siang Wang, Ying-Hui Lai, Yu Tsao, Hsiu-Wen Chang, and Hsin-Min Wang, “Audio-visual speech enhancement using multimodal deep convolutional neural networks,” IEEE Transactions on Emerging Topics in Computational Intelligence , vol. 2, no. 2, pp. 117–128, 2018

    2018

  10. [18]

    DNN driven speaker independent audio-visual mask estima- tion for speech separation,

    Mandar Gogate, Ahsan Adeel, Ricard Marxer, Jon Barker, and Amir Hussain, “DNN driven speaker independent audio-visual mask estima- tion for speech separation,” in Proc. Conference of the International Speech Communication Association (INTERSPEECH) , 2018, pp. 2723– 2727

    2018

  11. [19]

    Statistical speech enhancement based on probabilistic integration of variational autoencoder and non-negative matrix factorization,

    Yoshiaki Bando, Masato Mimura, Katsutoshi Itoyama, Kazuyoshi Yoshii, and Tatsuya Kawahara, “Statistical speech enhancement based on probabilistic integration of variational autoencoder and non-negative matrix factorization,” in Proc. IEEE International Conference on Acoustics,...

    2018

  12. [20]

    A variance modeling framework based on variational autoencoders for speech enhancement,

    Simon Leglaive, Laurent Girin, and Radu Horaud, “A variance modeling framework based on variational autoencoders for speech enhancement,” in Proc. IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2018, pp. 1–6

    2018

  13. [21]

    Bayesian multichannel speech enhancement with a deep speech prior,

    Kouhei Sekiguchi, Yoshiaki Bando, Kazuyoshi Yoshii, and Tatsuya Kawahara, “Bayesian multichannel speech enhancement with a deep speech prior,” in Proc. Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) , 2018, pp. 1233–1239

    2018

  14. [22]

    Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization,

    Simon Leglaive, Laurent Girin, and Radu Horaud, “Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization,” in Proc. IEEE International Con- ference on Acoustics, Speech, and Signal Processing (ICASSP) , 2019, pp. 101–105

    2019

  15. [23]

    Speech enhancement with variational autoencoders and alpha-stable distributions,

    Simon Leglaive, Umut S ¸ims ¸ekli, Antoine Liutkus, Laurent Girin, and Radu Horaud, “Speech enhancement with variational autoencoders and alpha-stable distributions,” in Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019, pp. 541–545

    2019

  16. [24]

    A statistically principled and computationally efficient approach to speech enhancement using variational autoencoders,

    Manuel Pariente, Antoine Deleforge, and Emmanuel Vincent, “A statistically principled and computationally efficient approach to speech enhancement using variational autoencoders,” in Proc. Conference of the International Speech Communication Association (INTERSPEECH) , 2019

    2019

  17. [25]

    Learning structured output representation using deep conditional generative models,

    Kihyuk Sohn, Honglak Lee, and Xinchen Yan, “Learning structured output representation using deep conditional generative models,” in Proc. Advances in Neural Information Processing Systems (NIPS), 2015, pp. 3483–3491

    2015

  18. [26]

    Multichannel speech enhancement based on time-frequency masking using subband long short-term memory,

    Xiaofei Li and Radu Horaud, “Multichannel speech enhancement based on time-frequency masking using subband long short-term memory,” in IEEE Workshop on Applications of Signal Processing to Audio and Acoustics. IEEE, 2019, pp. 298–302

    2019

  19. [27]

    NTCD-TIMIT: A new database and baseline for noise-robust audio-visual speech recognition,

    Ahmed Hussen Abdelaziz, “NTCD-TIMIT: A new database and baseline for noise-robust audio-visual speech recognition,” inProc. Conference of the International Speech Communication Association (INTERSPEECH) , 2017, pp. 3752–3756

    2017

  20. [28]

    An audio- visual corpus for speech perception and automatic speech recognition,

    Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao, “An audio- visual corpus for speech perception and automatic speech recognition,” J. Acoustical Society of America , vol. 120, no. 5, pp. 2421–2424, 2006

    2006

  21. [29]

    Suppression of acoustic noise in speech using spectral subtraction,

    Steven Boll, “Suppression of acoustic noise in speech using spectral subtraction,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 27, no. 2, pp. 113–120, 1979

    1979

  22. [30]

    Enhancement and bandwidth compression of noisy speech,

    Jae Soo Lim and Alan V Oppenheim, “Enhancement and bandwidth compression of noisy speech,” Proceedings of the IEEE , vol. 67, no. 12, pp. 1586–1604, 1979

    1979

  23. [31]

    Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator,

    Yariv Ephraim and David Malah, “Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator,” IEEE Transactions on Acoustics, Speech, and Signal Processing , vol. 32, no. 6, pp. 1109–1121, 1984

    1984

  24. [32]

    Speech enhancement based on minimum mean-square error estimation and supergaussian priors,

    Rainer Martin, “Speech enhancement based on minimum mean-square error estimation and supergaussian priors,”IEEE Transactions on Speech and Audio Processing , vol. 13, no. 5, pp. 845–856, 2005

    2005

  25. [33]

    Minimum mean-square error estimation of discrete Fourier coefficients with generalized Gamma priors,

    Jan Erkelens, Richard Hendriks, Richard Heusdens, and Jesper Jensen, “Minimum mean-square error estimation of discrete Fourier coefficients with generalized Gamma priors,” IEEE Transactions on Audio, Speech, and Language Processing , vol. 15, no. 6, pp. 1741–1752, 2007

    2007

  26. [34]

    Speech enhancement using a minimum mean-square error log-spectral amplitude estimator,

    Yariv Ephraim and David Malah, “Speech enhancement using a minimum mean-square error log-spectral amplitude estimator,” IEEE Transactions on Acoustics, Speech, and Signal Processing , vol. 33, no. 2, pp. 443–445, 1985. 13

    1985

  27. [35]

    Speech enhancement for non- stationary noise environments,

    Israel Cohen and Baruch Berdugo, “Speech enhancement for non- stationary noise environments,” Signal processing, vol. 81, no. 11, pp. 2403–2418, 2001

    2001

  28. [36]

    Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis,

    C ´edric F ´evotte, Nancy Bertin, and Jean-Louis Durrieu, “Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis,” Neural computation , vol. 21, no. 3, pp. 793–830, 2009

    2009

  29. [37]

    Speech denoising using nonnegative matrix factorization with priors,

    Kevin Wilson, Bhiksha Raj, Paris Smaragdis, and Ajay Divakaran, “Speech denoising using nonnegative matrix factorization with priors,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Las Vegas, USA, 2008, pp. 4029–4032

    2008

  30. [38]

    Phoneme-dependent NMF for speech enhancement in monaural mixtures,

    Bhiksha Raj, Rita Singh, and Tuomas Virtanen, “Phoneme-dependent NMF for speech enhancement in monaural mixtures,” in Proc. Confer- ence of the International Speech Communication Association (INTER- SPEECH), 2011, pp. 1217–1220

    2011

  31. [39]

    Super- vised and unsupervised speech enhancement using nonnegative matrix factorization,

    Nasser Mohammadiha, Paris Smaragdis, and Arne Leijon, “Super- vised and unsupervised speech enhancement using nonnegative matrix factorization,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 21, no. 10, pp. 2140–2151, 2013

    2013

  32. [40]

    Supervised speech separation based on deep learning: An overview,

    DeLiang Wang and Jitong Chen, “Supervised speech separation based on deep learning: An overview,” IEEE Transactions on Audio, Speech, and Language Processing , vol. 26, no. 10, pp. 1702–1726, 2018

    2018

  33. [41]

    Speech enhancement based on deep denoising autoencoder.,

    Xugang Lu, Yu Tsao, Shigeki Matsuda, and Chiori Hori, “Speech enhancement based on deep denoising autoencoder.,” in Proc. Confer- ence of the International Speech Communication Association (INTER- SPEECH), 2013, pp. 436–440

    2013

  34. [42]

    A regression approach to speech enhancement based on deep neural networks,

    Yong Xu, Jun Du, Li-Rong Dai, and Chin-Hui Lee, “A regression approach to speech enhancement based on deep neural networks,” IEEE Transactions on Audio, Speech, and Language Processing , vol. 23, no. 1, pp. 7–19, 2015

    2015

  35. [43]

    SNR-aware convolutional neural network modeling for speech enhancement.,

    Szu-Wei Fu, Yu Tsao, and Xugang Lu, “SNR-aware convolutional neural network modeling for speech enhancement.,” in Proc. Conference of the International Speech Communication Association (INTERSPEECH) , 2016, pp. 3768–3772

    2016

  36. [44]

    Towards scaling up classification- based speech separation,

    Yuxuan Wang and DeLiang Wang, “Towards scaling up classification- based speech separation,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 21, no. 7, pp. 1381–1390, 2013

    2013

  37. [45]

    On training targets for supervised speech separation,

    Yuxuan Wang, Arun Narayanan, and DeLiang Wang, “On training targets for supervised speech separation,” IEEE/ACM transactions on audio, speech, and language processing, vol. 22, no. 12, pp. 1849–1858, 2014

    2014

  38. [46]

    Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ASR,

    Felix Weninger, Hakan Erdogan, Shinji Watanabe, Emmanuel Vincent, Jonathan Le Roux, John R Hershey, and Bj ¨orn Schuller, “Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ASR,” in Proc. International Conference on Latent Variable Anal...

    2015

  39. [47]

    Semi-supervised learning with deep generative models,

    Diederik. P. Kingma, Danilo J. Rezende, Shakir Mohamedy, and Max Welling, “Semi-supervised learning with deep generative models,” in Adv. Neural Information Processing Systems (NIPS) , 2014, pp. 3581– 3589

    2014

  40. [48]

    Su- pervised determined source separation with multichannel variational autoencoder,

    Hirokazu Kameoka, Li Li, Shota Inoue, and Shoji Makino, “Su- pervised determined source separation with multichannel variational autoencoder,” Neural Computation, vol. 31, no. 9, pp. 1–24, 2019

    2019

  41. [49]

    Fast MV AE: Joint separation and classification of mixed sources based on multichannel variational autoencoder with auxiliary classifier,

    Li Li, Hirokazu Kameoka, and Shoji Makino, “Fast MV AE: Joint separation and classification of mixed sources based on multichannel variational autoencoder with auxiliary classifier,” in Proc. IEEE In- ternational Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2...

    2019

  42. [50]

    Joint separation and dereverberation of reverberant mixtures with multichannel variational autoencoder,

    Shota Inoue, Hirokazu Kameoka, Li Li, Shogo Seki, and Shoji Makino, “Joint separation and dereverberation of reverberant mixtures with multichannel variational autoencoder,” in Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , 2019, pp. 96–100

    2019

  43. [51]

    Visually derived wiener filters for speech enhancement,

    Ibrahim Almajai and Ben Milner, “Visually derived wiener filters for speech enhancement,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 6, pp. 1642–1651, 2010

    2010

  44. [52]

    A Monte Carlo implementation of the EM algorithm and the poor man’s data augmentation algorithms,

    Greg C.G. Wei and Martin A. Tanner, “A Monte Carlo implementation of the EM algorithm and the poor man’s data augmentation algorithms,” Journal of the American statistical Association , vol. 85, no. 411, pp. 699–704, 1990

    1990

  45. [53]

    An introduction to variational methods for graphical models,

    Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul, “An introduction to variational methods for graphical models,” Machine learning, vol. 37, no. 2, pp. 183–233, 1999

    1999

  46. [54]

    Variational inference: A review for statisticians,

    David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe, “Variational inference: A review for statisticians,” Journal of the American Statistical Association, vol. 112, no. 518, pp. 859–877, 2017

    2017

  47. [55]

    End-to-end audiovisual speech recognition,

    Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Feipeng Cai, Geor- gios Tzimiropoulos, and Maja Pantic, “End-to-end audiovisual speech recognition,” in Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , 2018, pp. 6548–6552

    2018

  48. [56]

    β-vae: Learning basic visual concepts with a constrained variational framework,

    Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner, “β-vae: Learning basic visual concepts with a constrained variational framework,” in International Conference on Learning Representations (ICLR), 2017

    2017

  49. [57]

    Maximum likelihood from incomplete data via the EM algorithm,

    Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” Journal of the royal statistical society. Series B (Methodological) , vol. 39, no. 1, pp. 1–38, 1977

    1977

  50. [58]

    Robert and George Casella, Monte Carlo Statistical Methods, Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2005

    Christian P. Robert and George Casella, Monte Carlo Statistical Methods, Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2005

    2005

  51. [59]

    Algorithms for nonnegative matrix factorization with the β-divergence,

    C ´edric F ´evotte and J ´erˆome Idier, “Algorithms for nonnegative matrix factorization with the β-divergence,” Neural computation, vol. 23, no. 9, pp. 2421–2456, 2011

    2011

  52. [60]

    TIMIT acoustic phonetic continuous speech corpus,

    John S. Garofolo, Lori F. Lamel, William M. Fisher, Jonathan G. Fiscus, David S. Pallett, Nancy L. Dahlgren, and Victor Zue, “TIMIT acoustic phonetic continuous speech corpus,” in Linguistic data consortium , 1993

    1993

  53. [61]

    FaNT– filtering and noise adding tool,

    Hans-G ¨unter Hirsch, “FaNT– filtering and noise adding tool,” Tech. Rep., International Computer Science Institute, Niederrhein University of Applied Science, 2005

    2005

  54. [62]

    The Diverse Environments Multi-channel Acoustic Noise Database (DEMAND): A database of multichannel environmental noise recordings,

    Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent, “The Diverse Environments Multi-channel Acoustic Noise Database (DEMAND): A database of multichannel environmental noise recordings,” in Proc. International Congress on Acoustics , 2013

    2013

  55. [63]

    Super- vised and semi-supervised separation of sounds from single-channel mixtures,

    Paris Smaragdis, Bhiksha Raj, and Madhusudana Shashanka, “Super- vised and semi-supervised separation of sounds from single-channel mixtures,” in Proc. Int. Conf. Indep. Component Analysis and Signal Separation, 2007, pp. 414–421

    2007

  56. [64]

    A non-negative approach to semi-supervised separation of speech from noise with the use of tem- poral dynamics,

    Gautham J. Mysore and Paris Smaragdis, “A non-negative approach to semi-supervised separation of speech from noise with the use of tem- poral dynamics,” in 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2011

    2011

  57. [65]

    Adam: A method for stochastic optimization,

    Diederik P. Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR), 2015

    2015

  58. [66]

    Performance measurement in blind audio source separation,

    Emmanuel Vincent, R ´emi Gribonval, and C ´edric F´evotte, “Performance measurement in blind audio source separation,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 14, no. 4, pp. 1462–1469, 2006

    2006

  59. [67]

    Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,

    Antony W. Rix, John G. Beerends, Michael P. Hollier, and Andries P. Hekstra, “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” in Proc. IEEE International Conference on Acoustics, Speech, and Signal Pr...

    2001

  60. [68]

    An algorithm for intelligibility prediction of time–frequency weighted noisy speech,

    Cees H. Taal, Richard C. Hendriks, Richard Heusdens, and Jesper Jensen, “An algorithm for intelligibility prediction of time–frequency weighted noisy speech,” IEEE Trans. Audio, Speech, Language Process., vol. 19, no. 7, pp. 2125–2136, 2011

    2011

  61. [69]

    A deep generative model of speech complex spectrograms,

    Aditya Arie Nugraha, Kouhei Sekiguchi, and Kazuyoshi Yoshii, “A deep generative model of speech complex spectrograms,” in Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019, pp. 905–909

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.