Pith. sign in

REVIEW 3 major objections 4 minor 58 references

Reading to Listen at the Cocktail Party: Multi-Modal Speech Separation

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VoiceFormer claims that a single transformer-bottleneck network can separate a target speaker using text, lip movements, or both, with state-of-the-art results on LRS2 and LRS3.

desk verdict Text-conditioned separation is a genuinely new capability and the architecture is sound, but the state-of-the-art claim rests on custom test sets and mismatched baselines that need verification. read the letter →

arxiv 2501.01518 v1 pith:NSVGFUDY submitted 2025-01-02 eess.AS cs.SDeess.SP

Akam Rahimi , Triantafyllos Afouras , Andrew Zisserman This is my paper · ORCID
classification eess.AScs.SDeess.SP
keywords speechseparationenhancementtext-conditionedaudio-visualfusiontransformerbottleneckrawwaveformcocktailpartyeffectLRS2/LRS3
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents VoiceFormer, a speech separation and enhancement model that conditions on the text of the target utterance, the speaker's lip movements, or both simultaneously. The central claim is that a single transformer-bottleneck architecture can fuse these heterogeneous cues even when they are not temporally aligned, and that text alone is a workable conditioning signal for pulling one voice out of a mixture. The authors report state-of-the-art separation scores on LRS2 and LRS3, and show that adding text to video makes the model substantially more tolerant of missing or corrupted video frames and of audio-video synchronisation offsets. A sympathetic reader should care because this opens a practical route to target-speaker isolation in settings where transcripts or subtitles are available but clean video is not.

What carries the argument

The load-bearing mechanism is a U-Net audio encoder-decoder operating directly on raw waveforms, with a multi-layer Transformer encoder inserted as the bottleneck. Audio, video, and phoneme-sequence embeddings are each augmented with a positional encoding and a learnable modality encoding, concatenated along the time dimension, and processed with full cross-attention, so every audio token can attend to the corresponding video tokens and text tokens. The positional encodings record timestamps for audio and video and word order for text, which is what allows conditioning on asynchronous streams without explicit alignment and lets the Transformer exploit language structure. The audio-side Transformer outputs are decoded back into a cleaned waveform, and the whole network is trained end-to-end with an L1 loss against the clean target.

What would settle it

Evaluate VoiceFormer and the Table 4 baselines on a common, unfiltered LRS2/LRS3 test set with identical two-speaker mixtures; if the reported SDR lead shrinks or disappears, the state-of-the-art claim depends on the authors' custom filtering rather than on the architecture.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the textual content of an utterance is a usable and previously underexplored cue for speaker separation. An audio-plus-text model reaches SDR 13.1 on LRS2 and 14.1 on LRS3, close to the audio-plus-video model at 14.1 and 15.5, while the full audio-plus-video-plus-text model reaches 14.2 and 15.5 and is markedly more tolerant when video frames are masked. The same design also removes the need for synchronised or same-rate audio and video streams: the model retains high SDR with offsets up to 200 ms, and the attention maps show it learns cross-modal alignment implicitly. The authors conclude that lip movements are a stronger cue than text, but that text provides robustness with little downside.

Load-bearing premise

The state-of-the-art comparison assumes the authors' filtered, synthetically mixed LRS2/LRS3 test sets are directly comparable to the test conditions used by prior baselines, even though the paper removes samples with noise or multiple speakers and builds its own evaluation splits.

Editorial extensions

If this is right

  • Text-only conditioning separates two-speaker mixtures at SDR 13.1 (LRS2) and 14.1 (LRS3), confirming that content, not just voice or face identity, can drive separation.
  • Adding text to video raises SDR only slightly over video alone (14.2 vs 14.1 on LRS2) but makes the model markedly more tolerant of video frame masking.
  • The model keeps SDR above 12 with audio-video offsets up to 200 ms, while an LSTM-bottleneck baseline degrades sharply under the same misalignment.
  • With inconsistent text but correct video, the A+V+T model performs on par with video-only conditioning; with inconsistent video but correct text, it partially recovers instead of failing completely.
  • On the denoising task, all VoiceFormer variants match the audio-only Denoiser baseline, so the multi-modal machinery does not sacrifice standard enhancement performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension not tested in the paper is ASR-conditioned separation: pair a real, error-prone ASR transcript with the audio stream. The word-removal experiments suggest graceful degradation, but real ASR error patterns were not measured, so the actual drop remains an open question.
  • Because conditioning uses only phoneme order rather than timestamps, the same architecture could likely accept other ordered symbolic inputs, such as subtitles, lyrics, or captions, as separation cues; the paper does not claim this.
  • The 200 ms desync tolerance suggests the method is a candidate for hearing aids or teleconferencing systems where wireless video and audio arrive with variable delays; a user study with genuinely out-of-sync recordings would be the next test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces VoiceFormer, a U-Net with a transformer bottleneck that performs speech separation and enhancement by conditioning on raw audio jointly with visual lip movements, textual (phoneme) input, or both. The architecture concatenates modality-specific sequences in the transformer bottleneck with positional and modality encodings, allowing asynchronous and non-aligned conditioning. Experiments on synthetic mixtures from LRS2 and LRS3 report SDR improvements over prior audio-visual baselines, including 14.1/15.5 dB for audio-visual separation (A+V) and 13.1/14.1 dB for text-only conditioning (A+T). The paper also demonstrates robustness to audio-visual synchronization offsets and to missing or inconsistent modalities, and it ablates architecture choices (LSTM vs. transformer bottleneck, spectrogram vs. waveform input).

Significance. The proposed framework is genuinely novel in unifying conditioning on text, video, and audio in a single transformer-bottleneck architecture, and the text-conditioned separation results are a useful new capability. The robustness experiments (Figure 5, Table 3) are carefully designed and show a clear advantage of the transformer over an LSTM baseline under audio-visual misalignment. The internal ablations (Table 2) support the architectural choices. However, the headline claim of state-of-the-art performance on 'well-established benchmark datasets' is currently not supported by the evidence because the evaluation uses custom filtered test sets and unmatched baselines. If the authors provide a matched evaluation or appropriately scope the claim, the contribution would be solid.

major comments (3)
  1. [§4.2 (Data) and Abstract] The central SOTA claim is defined on test sets that are not the same benchmarks used by prior methods. Section 4.2 states that 140 hours of LRS2 and 1 hour of LRS3 were removed (retaining 57/197 and 439/440 hours), and that the authors constructed their own test sets of 2515 and 3229 samples. Since SDR, STOI, and PESQ depend on the mixture construction and utterance content, the numbers in Table 4 are not directly comparable to previously published results on the original LRS2/LRS3 splits. To support the Abstract's 'state-of-the-art on well-established benchmark datasets,' the authors must either evaluate on the same test splits used by the baselines or provide a matched protocol in which all models are trained and tested on identical filtered data.
  2. [§4.4, Table 4] The baseline comparisons are not matched. VisualVoice [21] is explicitly not fine-tuned on the synthetic two-speaker LRS2 training set (footnote †), and Lee et al. [25] is evaluated on a different test set (footnote ◦). The Denoiser [12] row reports metrics identical to the 'Noisy input' row, indicating it fails to separate the two speakers, which is expected for an enhancement-only model; this does not provide a meaningful comparison for speaker separation. Because the baselines are not trained or evaluated under identical conditions, the reported margins (e.g., 3.3 dB over VisualVoice on LRS2) cannot be attributed to the proposed architecture or conditioning strategy. The authors should retrain or fine-tune all baselines on the same filtered training data and evaluate on the same test set, or, if this is not feasible, remove the global SOTA claim and present the results as demonstrating the method's internal improvements.
  3. [§4.2 (Evaluation sequences) and Reproducibility] The paper does not release code, the list of removed sample IDs, or the exact scripts for constructing the synthetic mixtures and test splits. The filtering procedure ('a combination of diarization and background noise detection methods') is not specified precisely enough for independent reproduction. Without the exact test IDs and mixture parameters (e.g., SNR distribution, normalization strategy), the headline numbers in Table 4 cannot be independently verified. At minimum, the authors should provide a detailed description of the filtering pipeline, the test sample IDs, and the exact mixing recipe, and ideally release the evaluation code and pre-extracted features.
minor comments (4)
  1. [§4.3 (Implementation details)] The text states 'we use N = 3 layers and h = 8 heads, with a model size of 532' and then says 'embedding dimensions across all modalities is set to 768.' The relationship between 532 and 768 is unclear; please clarify what 'model size' refers to (e.g., hidden dimension, feed-forward dimension, or total parameter count).
  2. [§4.4 (Cross-modal attention)] The sentence 'The attention maps in Figure 3 reveal the correspondence between the audio tokens and the other modalities. attend to the features in the corresponding modalities.' contains a grammatical fragment. It should be rewritten as a single coherent sentence.
  3. [Table 3 and Table 1] The ordering of modality labels is inconsistent: Table 1 and Table 2 use 'A+V+T' while Table 3 uses 'A+T+V' in the first column. Please unify the notation throughout.
  4. [References] In Section 3.1, 'Similar to [13], we extract a representation of the noisy audio...' appears to refer to the Denoiser architecture [12] rather than the music source separation paper [13]. Please verify the citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the model's outputs are measured against held-out synthetic mixtures, and the architecture's components are used as inputs, not as justifications of the results.

full rationale

No load-bearing step in this paper reduces to its own input by construction. The learning objective (Eq. 5) is a direct L1 loss between the predicted waveform and a held-out clean target, with conditioning inputs (audio mixture, VTP visual features, phonemized text) fed as external signals; no parameter is fitted to the reported SDR/STOI/PESQ numbers and then renamed as a prediction. Self-citations are present—[9]/[2] supply the LRS2/LRS3 corpora, [36] supplies the frozen VTP visual backbone, and [1]/[4] supply baselines—but none is used to forbid alternatives or to justify the central claim; the visual backbone is a pretrained feature extractor and the datasets are standard external benchmarks, so these citations constitute reproducible inputs rather than circular support. The state-of-the-art comparison in Table 4 does rely on the authors' own filtered LRS2/LRS3 test splits and on baselines evaluated under different protocols (footnotes in Table 4 disclose that VisualVoice was not fine-tuned on the synthetic task and that Lee et al. numbers are on a different test set), but this is an evaluation-protocol/matching concern, not a circularity in the sense of Eq. X reducing to Eq. Y or a fitted quantity being presented as a prediction. The ablations and robustness experiments are internally coherent and are measured on the same held-out mixtures rather than derived from the model's own equations. Accordingly no circular step is identified, and the paper scores 0 under the circularity rubric; the benchmarking caveats are correctness risks, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central results rest on trained network weights and hand-selected hyperparameters; there is no closed-form derivation. The most consequential choices are the synthetic mixing protocol, the filtered test splits, and the offset augmentation range, all under the authors' control and not independently benchmarked.

free parameters (4)
  • Transformer depth and heads = N=3 layers, h=8 heads
    Hand-selected architecture hyperparameters; no search protocol is reported, so the specific values are likely chosen by validation performance.
  • Embedding dimension = 768
    Hand-set to match the U-Net encoder channel dimension; the paper notes it separately from the stated model size of 532.
  • Audio-visual offset augmentation range = -200 to 200 ms
    The claimed robustness to synchronization offsets is trained under this distribution, so the robustness result is in part a consequence of this chosen range.
  • Training sequence lengths = 4 s for A+V; 1-6 s for text-conditioned models
    Chosen by available sample lengths and batching convenience; the amount of temporal context available to the transformer depends on these values.
assumptions (4)
  • domain assumption Synthetic mixtures of independently normalized clips are a valid proxy for real cocktail-party conditions.
    Stated in Section 4.1: 'the domain gap between synthetic and real samples is small.' All training and evaluation uses synthetic mixtures, so real-world generalization is assumed rather than measured.
  • domain assumption LRS2/LRS3 transcripts are accurate and the Phonemizer correctly converts text to phonemes.
    Training and evaluation rely on text-to-phoneme conversion; no error analysis is given for transcript accuracy or phonemizer mistakes.
  • ad hoc to paper The cleaned and filtered test sets are representative of the original LRS2/LRS3 benchmarks.
    The paper removes samples with background noise or multiple speakers and builds its own test splits. This is an ad hoc protocol choice that affects comparability with prior work.
  • domain assumption A transformer with positional and modality encodings can implicitly align asynchronous streams without explicit alignment.
    This is the core architectural hypothesis. It is tested empirically in Figure 5, but not proven beyond the tested offset range and datasets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reading to Listen at the Cocktail Party: Multi-Modal Speech Separation." pith.science (2026). https://pith.science/paper/NSVGFUDY

@misc{pith2026250101518,
  author       = {Pith},
  title        = {Pith review of: Reading to Listen at the Cocktail Party: Multi-Modal Speech Separation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NSVGFUDY}},
  note         = {Machine review of arXiv:2501.01518}
}
read the original abstract

The goal of this paper is speech separation and enhancement in multi-speaker and noisy environments using a combination of different modalities. Previous works have shown good performance when conditioning on temporal or static visual evidence such as synchronised lip movements or face identity. In this paper, we present a unified framework for multi-modal speech separation and enhancement based on synchronous or asynchronous cues. To that end we make the following contributions: (i) we design a modern Transformer-based architecture tailored to fuse different modalities to solve the speech separation task in the raw waveform domain; (ii) we propose conditioning on the textual content of a sentence alone or in combination with visual information; (iii) we demonstrate the robustness of our model to audio-visual synchronisation offsets; and, (iv) we obtain state-of-the-art performance on the well-established benchmark datasets LRS2 and LRS3.

Figures

Figures reproduced from arXiv: 2501.01518 by the authors.

Figure 1
Figure 1. We propose VoiceFormer, a framework for multi-modal [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed multi-modal speech enhancement with transformers (VoiceFormer) architecture. it consists of a u [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Attention map visualisations of the first Transformer layer. The visualisations show the average score of the atten￾tion heads in the first multi-head attention layer of the transformer. Brighter colours indicate higher scores and brighter pixels on the same row indicate correspondence between modalities. Left: au￾dio and video correlation; and Right: audio and text correlation. Higher scores are given to the audio … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Experiments with missing information. sistent video input. To sum up our analysis of the behaviour of VoiceFormer, we conclude that the A+V+T model provides good robust￾ness to disruptions in the video inputs, which comes with virtually no risk; even if for some reason…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 48 canonical work pages

  1. [21]

    VisualV oice: Audio- Visual Speech Separation with Cross-Modal Consistency

    Ruohan Gao and Kristen Grauman. VisualV oice: Audio- Visual Speech Separation with Cross-Modal Consistency. In Proc. CVPR, 2021. 1, 2, 3, 7, 8

    work page 2021

  2. [25]

    Looking into your speech: Learning cross-modal affinity for audio-visual speech sepa- ration

    Jiyoung Lee, Soo-Whan Chung, Sunok Kim, Hong-Goo Kang, and Kwanghoon Sohn. Looking into your speech: Learning cross-modal affinity for audio-visual speech sepa- ration. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2021. 8

    work page 2021

  3. [12]

    Real time speech enhancement in the waveform domain

    Alexandre Defossez, Gabriel Synnaeve, and Yossi Adi. Real time speech enhancement in the waveform domain. In Inter- speech, 2020. 2, 3, 4, 5, 7, 8

    work page 2020

  4. [1]

    The conversation: Deep audio-visual speech enhance- ment

    Triantafyllos Afouras, Joon Son Chung, and Andrew Zisser- man. The conversation: Deep audio-visual speech enhance- ment. In INTERSPEECH, 2018. 1, 3, 4, 7, 8

    work page 2018

  5. [2]

    LRS3-TED: a large-scale dataset for visual speech recognition

    Triantafyllos Afouras, Joon Son Chung, and Andrew Zis- serman. LRS3-TED: a large-scale dataset for visual speech recognition. In arXiv preprint arXiv:1809.00496, 2018. 4

    arXiv 2018

  6. [3]

    My lips are concealed: Audio-visual speech enhance- ment through obstructions

    Triantafyllos Afouras, Joon Son Chung, and Andrew Zisser- man. My lips are concealed: Audio-visual speech enhance- ment through obstructions. In INTERSPEECH, 2019. 2, 3

    work page 2019

  7. [4]

    Self-supervised learning of audio-visual objects from video

    Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self-supervised learning of audio-visual objects from video. In Proc. ECCV, 2020. 3, 6, 7, 8

    work page 2020

  8. [5]

    Assael, Brendan Shillingford, Shimon Whiteson, and Nando de Freitas

    Yannis M. Assael, Brendan Shillingford, Shimon Whiteson, and Nando de Freitas. Lipnet: Sentence-level lipreading. arXiv:1611.01599, 2016. 2

    arXiv 2016

Show all 58 references
  1. [6]

    Phonemizer: Text to phones transcription for multiple languages in python

    Mathieu Bernard and Hadrien Titeux. Phonemizer: Text to phones transcription for multiple languages in python. Jour- nal of Open Source Software, 6:3958, 12 2021. 3

    2021

  2. [7]

    Audio-visual synchronisation in the wild

    Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Na- grani, Andrea Vedaldi, and Andrew Zisserman. Audio-visual synchronisation in the wild. In Proc. BMVC, 2021. 3

    2021

  3. [8]

    Deep attrac- tor network for single-microphone speaker separation

    Zhuo Chen, Yi Luo, and Nima Mesgarani. Deep attrac- tor network for single-microphone speaker separation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 246–250, 2017. 2

    2017

  4. [9]

    Lip reading sentences in the wild

    Joon Son Chung, Andrew Senior, Oriol Vinyals, and An- drew Zisserman. Lip reading sentences in the wild. In Proc. CVPR, 2017. 4

    2017

  5. [10]

    Lip reading in the wild

    Joon Son Chung and Andrew Zisserman. Lip reading in the wild. In Proc. ACCV, 2016. 2

    2016

  6. [11]

    Facefilter: Audio-visual speech separation using still images

    Soo-Whan Chung, Soyeon Choe, Joon Son Chung, and Hong-Goo Kang. Facefilter: Audio-visual speech separation using still images. arXiv preprint arXiv:2005.07074, 2020. 1, 2

    2005 arXiv

  7. [13]

    Alexandre D ´efossez, Nicolas Usunier, L ´eon Bottou, and Francis R. Bach. Music Source Separation in the Waveform Domain. CoRR, abs/1911.13254, 2019. 3, 8

    1911 arXiv

  8. [14]

    Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separa- tion

    Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separa- tion. arXiv preprint arXiv:1804.03619, 2018. 1, 3, 4

    2018 arXiv

  9. [15]

    Learning joint statistical models for audio- visual fusion and segregation

    John W Fisher III, Trevor Darrell, William T Freeman, and Paul A Viola. Learning joint statistical models for audio- visual fusion and segregation. In NeurIPS, 2000. 2

    2000

  10. [16]

    Seeing through noise: Visually driven speaker separa- tion and enhancement

    Aviv Gabbay, Ariel Ephrat, Tavi Halperin, and Shmuel Pe- leg. Seeing through noise: Visually driven speaker separa- tion and enhancement. In Proc. ICASSP, pages 3051–3055. IEEE, 2018. 3

    2018

  11. [17]

    Visual Speech Enhancement using Noise-Invariant Training

    Aviv Gabbay, Asaph Shamir, and Shmuel Peleg. Visual Speech Enhancement using Noise-Invariant Training. arXiv preprint arXiv:1711.08789, 2017. 3

    2017 arXiv

  12. [18]

    Tenen- baum, and Antonio Torralba

    Chuang Gan, Deng Huang, Hang Zhao, Joshua B. Tenen- baum, and Antonio Torralba. Music gesture for visual sound separation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10475–10484, 2020. 3

    2020

  13. [19]

    Learning to separate object sounds by watching unlabeled video

    Ruohan Gao, Rog ´erio Schmidt Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. CoRR, abs/1804.01665, 2018. 2

    2018 arXiv

  14. [20]

    Co-separating sounds of visual objects

    Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. arXiv preprint arXiv:1904.07750, 2019. 2

    1904 arXiv

  15. [22]

    Multi-modal multi-channel tar- get speech separation

    Rongzhi Gu, Shi-Xiong Zhang, Yong Xu, Lianwu Chen, Yuexian Zou, and Dong Yu. Multi-modal multi-channel tar- get speech separation. IEEE Journal of Selected Topics in Signal Processing, 14(3):530–541, 2020. 3

    2020

  16. [23]

    Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe

    John R. Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discriminative embeddings for segmentation and separation. In 2016 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 31–35, 2016. 2

    2016

  17. [24]

    Perceiver: Gen- eral perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zis- serman, Oriol Vinyals, and Joao Carreira. Perceiver: Gen- eral perception with iterative attention. arXiv preprint arXiv:2103.03206, 2021. 3

    2021 arXiv

  18. [26]

    Parameter efficient multimodal trans- formers for video representation learning

    Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, and Yale Song. Parameter efficient multimodal trans- formers for video representation learning. In Proc. ICLR,

  19. [27]

    Audiovisual trans- former with instance attention for audio-visual event local- ization

    Yan-Bo Lin and Yu-Chiang Frank Wang. Audiovisual trans- former with instance attention for audio-visual event local- ization. In Proc. ACCV, 2020. 3

    2020

  20. [28]

    Speaker- independent speech separation with deep attractor network

    Yi Luo, Zhuo Chen, and Nima Mesgarani. Speaker- independent speech separation with deep attractor network. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(4):787–796, 2018. 2

    2018

  21. [29]

    Attention in dichotic listening: Affective cues and the influence of instructions

    Neville Moray. Attention in dichotic listening: Affective cues and the influence of instructions. Quarterly Journal of Experimental Psychology, 11(1):56–60, 1959. 1

    1959

  22. [30]

    Attention bottlenecks for multimodal fusion

    Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. NeurIPS, 2021. 3

    2021

  23. [31]

    Andrew Owens and Alexei A. Efros. Audio-visual scene analysis with self-supervised multisensory features. Proc. ECCV, 2018. 1, 3

    2018

  24. [32]

    Sanjeel Parekh, Slim Essid, Alexey Ozerov, Ngoc Q. K. Duong, Patrick P ´erez, and Ga ¨el Richard. Motion informed audio source separation. In 2017 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 6–10, 2017. 3

    2017

  25. [33]

    Audio-visual object localization and separation using low- rank and sparsity

    Jie Pu, Yannis Panagakis, Stavros Petridis, and Maja Pantic. Audio-visual object localization and separation using low- rank and sparsity. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2901–2905, 2017. 2

    2017

  26. [34]

    mir eval: A transparent implementation of common mir metrics

    Colin Raffel, Brian Mcfee, Eric Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, and Daniel Ellis. mir eval: A transparent implementation of common mir metrics. In Pro- ceedings - 15th International Society for Music Information Retrieval Conference (ISMIR 2014), 10 2014. 5

    2014

  27. [35]

    Interspeech 2021 deep noise suppression challenge

    Chandan KA Reddy, Harishchandra Dubey, Kazuhito Koishida, Arun Nair, Vishak Gopal, Ross Cutler, Sebastian Braun, Hannes Gamper, Robert Aichner, and Sriram Srini- vasan. Interspeech 2021 deep noise suppression challenge. In INTERSPEECH, 2021. 4

    2021

  28. [36]

    Visual keyword spotting with attention

    Prajwal K Renukanand, Liliane Momeni, Triantafyllos Afouras, and Andrew Zisserman. Visual keyword spotting with attention. In Proc. BMVC, 2021. 3, 5

    2021

  29. [37]

    Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of tele- phone networks and codecs

    Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of tele- phone networks and codecs. In Proc. ICASSP, volume 2, pages 749–752. IEEE, 2001. 5

    2001

  30. [38]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In International Conference on Medical image com- puting and computer-assisted intervention , pages 234–241. Springer, 2015. 2

    2015

  31. [39]

    Self-supervised audio-visual co-segmentation

    Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh Mc- Dermott, and Antonio Torralba. Self-supervised audio-visual co-segmentation. In Proc. ICASSP, pages 2357–2361. IEEE,

  32. [40]

    Audio-visual speech enhancement using conditional variational auto-encoders

    Mostafa Sadeghi, Simon Leglaive, Xavier Alameda-Pineda, Laurent Girin, and Radu Horaud. Audio-visual speech enhancement using conditional variational auto-encoders. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1788–1800, 2020. 3

    2020

  33. [41]

    Seeing to hear better: evidence for early audio- visual interactions in speech identification

    Jean-Luc Schwartz, Fr ´ed´eric Berthommier, and Christophe Savariaux. Seeing to hear better: evidence for early audio- visual interactions in speech identification. Cognition, 93(2):B69–B78, 2004. 1

    2004

  34. [42]

    Combining residual networks with lstms for lipreading

    Themos Stafylakis and Georgios Tzimiropoulos. Combining residual networks with lstms for lipreading. In Interspeech,

  35. [43]

    An algorithm for intelligibility prediction of time- frequency weighted noisy speech

    Cees Taal, Richard Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intelligibility prediction of time- frequency weighted noisy speech. IEEE Transactions on Au- dio, Speech and Language Processing, 2011. 5

    2011

  36. [44]

    Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing

    Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing. In Proc. ECCV, 2020. 3

    2020

  37. [45]

    Into the wild with audioscope: Unsupervised audio-visual separa- tion of on-screen sounds

    Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Her- shey, Tal Remez, Daniel PW Ellis, and John R Hershey. Into the wild with audioscope: Unsupervised audio-visual separa- tion of on-screen sounds. arXiv preprint arXiv:2011.01143,

    2011 arXiv

  38. [46]

    Efthymios Tzinis, Scott Wisdom, Tal Remez, and John R. Hershey. Improving on-screen sound separation for open domain videos with audio-visual self-attention. ArXiv, abs/2106.09669, 2021. 2

    2021 arXiv

  39. [47]

    BSS EV AL toolbox user guide

    Emmanuel Vincent, R ´emi Gribonval, and C ´edric F ´evotte. BSS EV AL toolbox user guide. IRISA Technical Report

  40. [48]

    Supervised speech sep- aration based on deep learning: An overview

    DeLiang Wang and Jitong Chen. Supervised speech sep- aration based on deep learning: An overview. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 26(10):1702–1726, 2018. 2

    2018

  41. [49]

    V oicefilter: Tar- geted voice separation by speaker-conditioned spectrogram masking

    Quan Wang, Hannah Muckenhirn, Kevin Wilson, Prashant Sridhar, Zelin Wu, John Hershey, Rif A Saurous, Ron J Weiss, Ye Jia, and Ignacio Lopez Moreno. V oicefilter: Tar- geted voice separation by speaker-conditioned spectrogram masking. In Interspeech, 2018. 1, 2

    2018

  42. [50]

    Combining spectral and spatial features for deep learning based blind speaker separation

    Zhong-Qiu Wang and DeLiang Wang. Combining spectral and spatial features for deep learning based blind speaker separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(2):457–468, 2019. 2

    2019

  43. [51]

    Time domain audio visual speech separation

    Jian Wu, Yong Xu, Shi-Xiong Zhang, Lian-Wu Chen, Meng Yu, Lei Xie, and Dong Yu. Time domain audio visual speech separation. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 667–673, 2019. 3

    2019

  44. [52]

    Multilevel language and vision integration for text-to-clip retrieval

    Huijuan Xu, Kun He, Bryan A Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko. Multilevel language and vision integration for text-to-clip retrieval. In AAAI, 2019. 3

    2019

  45. [53]

    Recursive visual sound separation using minus-plus net

    Xudong Xu, Bo Dai, and Dahua Lin. Recursive visual sound separation using minus-plus net. In2019 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV) , pages 882– 891, 2019. 2

    2019

  46. [54]

    Permutation invariant training of deep models for speaker-independent multi-talker speech separation

    Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jensen Jensen. Permutation invariant training of deep models for speaker-independent multi-talker speech separation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 241–245, 2017. 2

    2017

  47. [55]

    To find where you talk: Temporal sentence localization in video with attention based location regression

    Yitian Yuan, Tao Mei, and Wenwu Zhu. To find where you talk: Temporal sentence localization in video with attention based location regression. In AAAI, 2019. 3

    2019

  48. [56]

    The sound of motions

    Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Tor- ralba. The sound of motions. Proc. ICCV, 2019. 3

    2019

  49. [57]

    The sound of pixels

    Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl V on- drick, Josh McDermott, and Antonio Torralba. The sound of pixels. arXiv preprint arXiv:1804.03160, 2018. 2

    2018 arXiv

  50. [1706]

    http://www.irisa.fr/metiss/bss eval/., 2005. 5

    2005

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.