Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Online Audio-Visual Autoregressive Speaker Extraction

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An autoregressive acoustic cue adds 0.9 dB of SI-SNR improvement to an online audio-visual speaker extractor whose visual frontend has only 0.1 million parameters.

desk verdict Solid incremental AVSE paper with an overstated headline gain: the acoustic encoder adds ~0.5 dB once the hybrid loss is accounted for. read the letter →

arxiv 2506.01270 v1 pith:Q6JVZKAO submitted 2025-06-02 eess.AS cs.SD

Zexu Pan , Wupeng Wang , Shengkui Zhao , Chong Zhang , Kun Zhou , Yukun Ma , Bin Ma This is my paper
classification eess.AScs.SD
keywords audio-visualspeakerextractiononlinestreaminglightweightvisualfrontenddepth-wiseseparableconvolutionautoregressiveacousticencodertargetswitchingSI-SNRimprovementLRS3
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a streaming audio-visual speaker extraction system that separates one target speaker from a mixture in real time, conditioned on both the target's lip movements and the network's own previously extracted speech. The central claim is that a very small visual frontend built from depth-wise separable convolutions matches the accuracy of much larger ResNet18- and ShuffleNet-based frontends on the LRS3 benchmark, while using roughly 0.1 million parameters and 2.1 GMACs per second. Adding a lightweight autoregressive acoustic encoder that summarizes past extracted frames as frame-level conditioning improves SI-SNRi by 0.9 dB over the non-autoregressive system. The paper also introduces the first study of a switching target-speaker scenario, reporting that extraction quality remains equal before and after the switch, unlike a comparison system that aggregates past information into a single embedding.

What carries the argument

The arguments rest on two lightweight conditioning pathways feeding a SkiM speaker extractor. The visual pathway, BlazeNet64, uses a causal 3D convolution followed by depth-wise separable 2D convolutions to encode lip images, replacing the much heavier ResNet18-style encoder. The acoustic pathway is an autoregressive encoder: extracted speech from past frames passes through a speech encoder, several 1D convolutions, and one LSTM layer to produce a per-frame embedding A(t) that is concatenated with the audio and visual embeddings at the extractor input, actively carrying 'momentum' from previously separated frames. Because the network cannot use its own future outputs during training, it uses the PARIS two-pass scheme, where a first forward pass without the acoustic encoder produces 'pseudo past extracted speech' that conditions the second pass. The hybrid loss L1+L2 pairs SI-SNR with a multi-resolution delta spectrum loss, which the authors argue regulates output energy better than SNR alone for this autoregressive loop.

What would settle it

Run the trained autoregressive model in genuine frame-by-frame streaming over long multi-talker clips and compare SI-SNRi with and without the acoustic encoder; if the gap is smaller than the 0.9 dB reported under the PARIS evaluation, or if it shrinks as clip length grows, the pseudo-autoregressive training proxy does not transfer. Equally, in the switching-target scenario, measure SI-SNRi on the first seconds after a switch; if the 'after' value drops below the fixed-target value by more than the 0.1 dB reported, the momentum reset claim fails.

Watch

Extended reading notes

Core claim

The authors set out to show that an online audio-visual speaker extractor can be made dramatically cheaper on the visual side and more accurate on the audio side at the same time. Their BlazeNet64 visual encoder, with only 0.1 million parameters and 2.1 GMACs per second, attains 9.1 dB SI-SNRi on LRS3 with both the SkiM and ConvTasNet audio backbones, matching or slightly beating the ResNet18 frontend while cutting computation by a large factor. The second claim is that a frame-level autoregressive acoustic encoder, trained with the PARIS two-pass pseudo-autoregressive procedure and a hybrid SI-SNR plus frequency-domain loss, adds 0.9 dB SI-SNRi (from 9.1 to 10.0 dB) at the cost of only 0.5 million extra parameters, and that the gain is not erased when the target speaker changes mid-clip (before/after SI-SNRi both 9.9 dB). Together these establish that both visual and acoustic conditioning for streaming extraction can be lightweight without sacrificing quality.

Load-bearing premise

The system is trained with a stand-in for its own past outputs: during training the acoustic encoder reads the first-pass output, while during real-time inference it reads its own accumulated previous frames, and the paper does not establish that those two distributions match closely enough for the 0.9 dB gain to survive in true streaming.

Editorial extensions

If this is right

  • Streaming audio-visual speaker extraction can run on-device with a visual precondition that is over 100 times smaller in parameters than the ResNet18 frontend while keeping the same SI-SNRi.
  • The 0.9 dB SI-SNRi gain from the acoustic encoder transfers to a second dataset pair: on VoxCeleb2 mixtures SI-SNRi rises from 5.6 to 6.4 dB and on TCD-TIMIT from 9.5 to 11.0 dB.
  • Frame-level acoustic conditioning preserves performance across a mid-utterance target switch, with before and after SI-SNRi both 9.9 dB, whereas the ConvTasNet-based system drops 0.7 dB after the switch.
  • Lowering the visual frame rate to 12.5 FPS costs 2.8 dB SI-SNRi, and 5 FPS collapses to 0.0 dB, so the 25 FPS visual stream remains necessary despite the small visual network.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the PARIS proxy is faithful, the same pseudo-autoregressive trick could be applied to other online conditioning signals, such as brain or voice-print embeddings, whenever the network has access to its own past outputs at inference.
  • The before/after equality at 9.9 dB suggests that frame-level acoustic embeddings reset extraction momentum quickly; a direct comparison against NeuroHeed's single-embedding aggregation on the same switching setup would test whether global speaker embeddings are the cause of slower re-focusing.
  • Because the visual encoder is trained from scratch without pretraining, the BlazeNet64 design likely transfers to other audiovisual tasks with tight compute budgets, such as on-device audio-visual speech enhancement; this is an extension the paper does not evaluate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an online audio-visual speaker extraction system with two new components: a lightweight visual encoder (BlazeNet64) based on depth-wise separable convolutions, and a lightweight autoregressive acoustic encoder that conditions the extractor on past extracted speech. The paper also introduces and evaluates a switching-target scenario. Experiments on LRS3, VoxCeleb2, and TCD-TIMIT show that the visual encoder matches or exceeds prior visual frontends (ResNet18, ShuffleNetV2) at a fraction of the parameter and MAC counts, and that the full system with the acoustic encoder attains the best SI-SNRi in most conditions. The code is released.

Significance. The visual frontend result is a concrete efficiency contribution: replacing an 11.2-million-parameter, 12.9-GMAC ResNet18 encoder with a 0.1-million-parameter, 2.1-GMAC BlazeNet64 degrades SI-SNRi by only 0.1 dB on LRS3 and even improves it on VoxCeleb2/TCD-TIMIT. This is a strong, well-supported finding. The acoustic encoder is a reasonable idea, and the study of target switching is a useful scenario that the community has largely ignored. However, the paper's central quantitative claims about the acoustic encoder ('additional 0.9 dB') and about its momentum robustness are not supported by the controlled ablations presented in Table 3, and the train/inference mismatch in the PARIS-style training (Section 2.2) is left unexamined. These issues are load-bearing for the paper's main narrative, though they are fixable with re-analysis and additional experiments.

major comments (4)
  1. [Abstract, Section 1, Table 3] The claim that the autoregressive acoustic encoder provides an 'additional 0.9 dB gain' is not supported by the paper's own ablation. System 12 (acoustic encoder, L1+L2) achieves 10.0 dB SI-SNRi and System 6 (no acoustic encoder, L_SI-SNR) achieves 9.1 dB, but these systems differ in both the acoustic encoder and the loss function. The appropriate control is System 13 (no acoustic encoder, L1), which reaches 9.5 dB; hence the acoustic encoder contributes only 10.0 - 9.5 = 0.5 dB, while the hybrid loss contributes 0.4 dB. The abstract and introduction should be corrected to attribute 0.5 dB to the acoustic encoder, or the experiments should be re-run with a matched loss condition.
  2. [Section 2.2, Table 3] The PARIS training strategy introduces a train/inference distribution mismatch that is not addressed. During training, the acoustic encoder receives the first-pass network output as 'pseudo past extracted speech'; during streaming inference, it receives its own accumulated output from previous frames. If the pseudo-past distribution differs materially from the self-generated past distribution, the reported gains (0.5 dB in the controlled ablation) may not transfer to real online use. The paper should either report a streaming evaluation where the acoustic encoder is fed its own prior outputs, or include an analysis that quantifies the distribution shift and its effect on SI-SNRi.
  3. [Section 4.1, Table 2, Section 4.2] The claim that the acoustic encoder's 'momentum is robust against the change in attention' is not demonstrated. System 6, without the acoustic encoder, already shows identical SI-SNRi before and after the target switch (8.9/8.9 dB), so the robustness is a property of the AV-SkiM backbone rather than something introduced by the acoustic encoder. To support the momentum-reset claim, the paper should compare the 'before' vs 'after' gap for System 6 and System 12 directly, and ideally show that the acoustic encoder does not hurt adaptation relative to a non-autoregressive system. As reported, the equality of 9.9/9.9 dB in System 12 is not evidence of a novel robustness mechanism.
  4. [General experimental methodology] No error bars, significance tests, or multiple-seed runs are reported anywhere in Tables 2-4. Given that the key acoustic-encoder gain shrinks from 0.9 dB to 0.5 dB under proper ablation, and given typical run-to-run variance in neural speech separation training, the reader cannot assess whether any of the differences attributed to the acoustic encoder are statistically reliable. The authors should report at least two or three seeds for the main systems (6, 11, 12, 13) and provide mean and standard deviation, or otherwise justify that single-run differences of 0.5 dB are meaningful.
minor comments (5)
  1. [Abstract, Table 1] The abstract says '2.1 MACs per second of processing,' but Table 1 reports MACs in billions (G). The text should say '2.1 GMACs per second' or '2.1 billion MACs per second.'
  2. [Section 2.2] There is a typo: 'fist pass' should be 'first pass' in the sentence defining L1 and L2.
  3. [Section 2.1 and Figure 1] The notation uses 'A(t)' for the acoustic embedding and 'V(t)' for the visual embedding, but the text later refers to 'A VSE' and 'A V-' prefixes; the spacing in 'A VSE' is inconsistent (e.g., 'A VSE' vs 'AVSE' in 'online A VSE'). Article copyediting should normalize these terms.
  4. [Table 3] The table omits a row for System 6 with the hybrid loss L1 but without the acoustic encoder; System 13 provides this condition, but the caption does not explicitly state that System 13 is the loss-matched control for isolating the acoustic encoder. Adding a sentence to the caption would help readers reproduce the 0.5 dB attribution.
  5. [Section 4.1] For systems 9 and 10 (reduced visual frame rates), the paper says performance 'drops significantly,' but System 10 at 5 FPS yields 0.0 dB SI-SNRi, which appears to be a total failure rather than a gradual degradation. A brief explanation of why 5 FPS leads to zero improvement (e.g., the visual signal is too sparse to provide any useful conditioning) would improve clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical comparisons, not derivations from fitted inputs.

full rationale

The paper makes no first-principles derivation; every load-bearing claim is an experimental SI-SNRi comparison on LRS3, VoxCeleb2, and TCD-TIMIT mixtures. The autoregressive acoustic encoder is evaluated by ablations (Table 3), and the visual frontend is compared against ResNet18 and ShuffleNetV2 baselines (Table 2). The training procedure adopts PARIS [28] and a hybrid loss from [33], both from the authors' prior work, but these are used as fixed implementation choices and are independently evaluated by the controlled systems (e.g., System 11 vs 12); they do not define the reported metric. The abstract's attribution of the full 0.9 dB gain to the acoustic encoder is not the best-controlled reading of Table 3 — System 13 isolates 0.5 dB for the encoder and 0.4 dB for the loss — but this is an over-attribution/statistical-support issue, not a reduction of an output to an input by construction. Similarly, the switching-robustness claim is an observed before/after SI-SNRi comparison rather than a derived result. No fitted parameter is renamed as a prediction and no equation is equivalent to its own input. Circularity score 0.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a hand-designed architecture and training recipe: the SkiM hyperparameters (Section 2.1), the loss weights 0.25 and 0.75 (Section 2.2), and the simulated mixture protocol (Section 3.1). No entity is invented, but the domain assumptions about LRS3-style simulations and the PARIS pseudo-past proxy are load-bearing.

free parameters (8)
  • SkiM LSTM hidden size = 384
    Hand-set in Section 2.1; not derived.
  • SkiM LSTM layers = 3
    Hand-set in Section 2.1.
  • SkiM non-overlapping segment size = 50
    Hand-set in Section 2.1.
  • Speech encoder channels, kernel, stride = 128, 16, 8
    Hand-set in Section 2.1.
  • Hybrid loss weights = 0.25 in L1, 0.75 in L2
    Hand-set in Section 2.2; the 0.9 dB versus 0.5 dB attribution depends on the loss formulation.
  • Mixture SNR range = 10 to -10 dB
    Randomly assigned in Section 3.1; determines task difficulty.
  • Training epochs, batch size, initial learning rate = 150, 16, 0.001
    Hand-set optimization choices in Section 3.2; could affect final SI-SNRi.
  • BlazeNet64 depth and channel widths = Not fully specified in text
    Architecture hyperparameters are hand-designed, motivated by BlazeFace; the paper reports only parameter and MAC counts, not the full layer table.
assumptions (4)
  • domain assumption Simulated two-speaker mixtures from LRS3 with random SNR from 10 to -10 dB are representative of real streaming multi-party audio-visual speaker extraction conditions.
    Section 3.1; all headline results and switching-target conclusions are based on these synthetic mixtures.
  • domain assumption The first-pass output in PARIS training is a valid proxy for past extracted speech during online autoregressive inference.
    Section 2.2; no direct validation of this train-inference mismatch is provided.
  • domain assumption Lip-region video at 25 FPS is synchronized with the target speech and provides sufficient identity information for switching targets.
    Section 3.1; the visual cue is the only target identity signal in the switching scenario.
  • domain assumption Objective metrics such as SI-SNRi, PESQi, and STOIi on simulated mixtures are sufficient to establish extraction quality and robustness.
    Section 4; no listening tests or real-device latency measurements are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Online Audio-Visual Autoregressive Speaker Extraction." pith.science (2026). https://pith.science/paper/Q6JVZKAO

@misc{pith2026250601270,
  author       = {Pith},
  title        = {Pith review of: Online Audio-Visual Autoregressive Speaker Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q6JVZKAO}},
  note         = {Machine review of arXiv:2506.01270}
}
read the original abstract

This paper proposes a novel online audio-visual speaker extraction model. In the streaming regime, most studies optimize the audio network only, leaving the visual frontend less explored. We first propose a lightweight visual frontend based on depth-wise separable convolution. Then, we propose a lightweight autoregressive acoustic encoder to serve as the second cue, to actively explore the information in the separated speech signal from past steps. Scenario-wise, for the first time, we study how the algorithm performs when there is a change in focus of attention, i.e., the target speaker. Experimental results on LRS3 datasets show that our visual frontend performs comparably to the previous state-of-the-art on both SkiM and ConvTasNet audio backbones with only 0.1 million network parameters and 2.1 MACs per second of processing. The autoregressive acoustic encoder provides an additional 0.9 dB gain in terms of SI-SNRi, and its momentum is robust against the change in attention.

Figures

Figures reproduced from arXiv: 2506.01270 by the authors.

Figure 1
Figure 1. Our proposed online audio-visual autoregressive speaker extraction network. The middle section introduces the network components and data flow, the top section showcases our proposed lightweight visual encoder, and the bottom sec￾tion depicts our proposed lightweight autoregressive acoustic encoder. Each colored layer’s parameters are specified in terms of kernel size, layer type, output channel size, and stride. ma… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 43 canonical work pages

  1. [1]

    Online Audio-Visual Autoregressive Speaker Extraction

    Introduction Real-world speech signals are often mixed with interfering speech and noise signals. Human brains excel at focusing on an interested speech signal, i.e., target speech, while filtering out the rest, known as auditory attention [1]. Equipping machines with such auditory selective attention is crucial for speech ap- plications such as automatic...

    work page Pith review arXiv 2025

  2. [2]

    pseudo past extracted speech

    Proposed network Let x(τ ) be a multi-talker mixture speech signal 1, consisting of the target speech signal s(τ ) and interference speech signal b(τ ), the A VSE networkf (·) estimates the target speech signal ˆs(τ ) to approximate s(τ ), conditioned on the visual recording of the target speaker v(t): ˆs(τ ) =f (x(τ ), v(t)) (1) In this work, we addition...

    work page

  3. [3]

    Dataset We mainly use the Lip Reading Sentences 3 (LRS3) dataset to validate our proposed method in this work [29], which is widely used in many A VSE studies [34–36]

    Experimental setup 3.1. Dataset We mainly use the Lip Reading Sentences 3 (LRS3) dataset to validate our proposed method in this work [29], which is widely used in many A VSE studies [34–36]. The speech signal is avail- able at 16kHz, and video is available at 25 frames per second. We study the following three scenarios in this paper: 1) Two speakers spea...

    work page

  4. [4]

    All improve- ments are calculated relative to the unprocessed multi-talker speech signals

    Results To evaluate the quality of extracted speech, we use several met- rics: the improvement in SI-SNR (SI-SNRi) [32], the improve- ment in SNR (SNRi) [38], the improvement in Perceptual Eval- uation of Speech Quality (PESQi) [39], and the improvement in Short-Term Objective Intelligibility (STOIi) [40]. All improve- ments are calculated relative to the...

    work page

  5. [5]

    The proposed visual en- coder, with its lightweight design and efficient processing, pro- vides a competitive alternative to the more complex visual en- coder

    Conclusion In conclusion, this work presents a significant advancement in online audio-visual speaker extraction by addressing both com- putational efficiency and performance. The proposed visual en- coder, with its lightweight design and efficient processing, pro- vides a competitive alternative to the more complex visual en- coder. The novel acoustic en...

    work page

  6. [6]

    Some experiments on the recognition of speech, with one and with two ears,

    E. C. Cherry, “Some experiments on the recognition of speech, with one and with two ears,” J. Acoust. Soc. Am. , vol. 25, no. 5, pp. 975–979, 1953

    work page 1953

  7. [7]

    Restoring speaking lips from occlusion for audio-visual speech recognition,

    J. Wang, Z. Pan, M. Zhang, R. T. Tan, and H. Li, “Restoring speaking lips from occlusion for audio-visual speech recognition,” in Proc. AAAI, vol. 38, 2024

    work page 2024

  8. [8]

    Deep clus- tering: Discriminative embeddings for segmentation and separa- tion,

    J. R. Hershey, Z. Chen, J. Le Roux, and S. Watanabe, “Deep clus- tering: Discriminative embeddings for segmentation and separa- tion,” in Proc. ICASSP, 2016, pp. 31–35

    work page 2016

Show all 47 references
  1. [9]

    Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation,

    Y . Luo and N. Mesgarani, “Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 27, no. 8, pp. 1256–1266, 2019

    2019

  2. [10]

    Dual-Path RNN: Efficient long sequence modeling for time-domain single-channel speech sepa- ration,

    Y . Luo, Z. Chen, and T. Yoshioka, “Dual-Path RNN: Efficient long sequence modeling for time-domain single-channel speech sepa- ration,” in Proc. ICASSP, 2020, pp. 46–50

    2020

  3. [11]

    TF-GridNet: Making time-frequency domain models great again for monaural speaker separation,

    Z.-Q. Wang, S. Cornell, S. Choi, Y . Lee, B.-Y . Kim, and S. Watan- abe, “TF-GridNet: Making time-frequency domain models great again for monaural speaker separation,” in Proc. ICASSP, 2023

    2023

  4. [12]

    Mossformer2: Combining transformer and rnn-free recurrent network for enhanced time- domain monaural speech separation,

    S. Zhao, Y . Ma, C. Ni, C. Zhang, H. Wang, T. H. Nguyen, K. Zhou, J. Yip, D. Ng, and B. Ma, “Mossformer2: Combining transformer and rnn-free recurrent network for enhanced time- domain monaural speech separation,” in Proc. ICASSP, 2024

    2024

  5. [13]

    V oice- Filter: Targeted voice separation by speaker-conditioned spectro- gram masking,

    Q. Wang, H. Muckenhirn, K. Wilson, P. Sridhar, Z. Wu, J. R. Her- shey, R. A. Saurous, R. J. Weiss, Y . Jia, and I. L. Moreno, “V oice- Filter: Targeted voice separation by speaker-conditioned spectro- gram masking,” in Proc. Interspeech, 2019, pp. 2728–2732

    2019

  6. [14]

    SpeakerBeam: Speaker aware neural network for target speaker extraction in speech mixtures,

    K. ˇZmol´ıkov´a, M. Delcroix, K. Kinoshita, T. Ochiai, T. Nakatani, L. Burget, and J. ˇCernock´y, “SpeakerBeam: Speaker aware neural network for target speaker extraction in speech mixtures,”IEEE J. Sel. Top. Signal Process., vol. 13, no. 4, pp. 800–814, 2019

    2019

  7. [15]

    SpEx: Multi-scale time domain speaker extraction network,

    C. Xu, W. Rao, E. S. Chng, and H. Li, “SpEx: Multi-scale time domain speaker extraction network,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 28, pp. 1370–1384, 2020

    2020

  8. [16]

    Looking to listen at the cock- tail party: a speaker-independent audio-visual model for speech separation,

    A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein, “Looking to listen at the cock- tail party: a speaker-independent audio-visual model for speech separation,” ACM Trans. Graph., vol. 37, no. 4, pp. 1–11, 2018

    2018

  9. [17]

    Scenario-aware audio-visual TF- Gridnet for target speech extraction,

    Z. Pan, G. Wichern, Y . Masuyama, F. G. Germain, S. Khu- rana, C. Hori, and J. Le Roux, “Scenario-aware audio-visual TF- Gridnet for target speech extraction,” in Proc. ASRU, 2023

    2023

  10. [18]

    Time domain audio visual speech separation,

    J. Wu, Y . Xu, S. Zhang, L. Chen, M. Yu, L. Xie, and D. Yu, “Time domain audio visual speech separation,” inProc. ASRU, 2019, pp. 667–673

    2019

  11. [19]

    Selective listening by synchro- nizing speech with lips,

    Z. Pan, R. Tao, C. Xu, and H. Li, “Selective listening by synchro- nizing speech with lips,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 30, pp. 1650–1664, 2022

    2022

  12. [20]

    FaceFilter: Audio-visual speech separation using still images,

    S.-W. Chung, S. Choe, J. S. Chung, and H.-G. Kang, “FaceFilter: Audio-visual speech separation using still images,” inProc. Inter- speech, 2020, pp. 3481–3485

    2020

  13. [21]

    Brain- informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception,

    E. Ceolini, J. Hjortkjær, D. D. Wong, J. O’Sullivan, V . S. Ragha- van, J. Herrero, A. D. Mehta, S.-C. Liu, and N. Mesgarani, “Brain- informed speech separation (BISS) for enhancement of target speaker in multitalker speech perception,” NeuroImage, vol. 223, p. 117282, 2020

    2020

  14. [22]

    Neu- roHeed: Neuro-steered speaker extraction using EEG signals,

    Z. Pan, M. Borsdorf, S. Cai, T. Schultz, and H. Li, “Neu- roHeed: Neuro-steered speaker extraction using EEG signals,” arXiv preprint arXiv:2307.14303, 2023

    2023 arXiv

  15. [23]

    NeuroHeed+: Improving neuro-steered speaker extraction with joint auditory attention detection,

    Z. Pan, G. Wichern, F. G. Germain, S. Khurana, and J. Le Roux, “NeuroHeed+: Improving neuro-steered speaker extraction with joint auditory attention detection,” in Proc. ICASSP, 2024, pp. 11 456–11 460

    2024

  16. [24]

    V oiceFilter-Lite: Streaming targeted voice separation for on-device speech recog- nition,

    Q. Wang, I. L. Moreno, M. Saglam, K. Wilson, A. Chiao, R. Liu, Y . He, W. Li, J. Pelecanos, M. Nika et al. , “V oiceFilter-Lite: Streaming targeted voice separation for on-device speech recog- nition,” Proc. Interspeech, pp. 2677–2681, 2020

    2020

  17. [25]

    Papez: Resource-efficient speech sepa- ration with auditory working memory,

    H. Oh, J. Yi, and Y . Lee, “Papez: Resource-efficient speech sepa- ration with auditory working memory,” inProc. ICASSP, 2023

    2023

  18. [26]

    SkiM: Skipping memory LSTM for low-latency real-time continuous speech separation,

    C. Li, L. Yang, W. Wang, and Y . Qian, “SkiM: Skipping memory LSTM for low-latency real-time continuous speech separation,” in Proc. ICASSP, 2022, pp. 681–685

    2022

  19. [27]

    Resource-efficient separation transformer,

    L. D. Libera, C. Subakan, M. Ravanelli, S. Cornell, F. Lepoutre, and F. Grondin, “Resource-efficient separation transformer,” in Proc. ICASSP, 2024

    2024

  20. [28]

    RT-LA-V ocE: Real- time low-SNR audio-visual speech enhancement,

    H. Chen, R. Mira, S. Petridis, and M. Pantic, “RT-LA-V ocE: Real- time low-SNR audio-visual speech enhancement,” Proc. Inter- speech, 2024

    2024

  21. [29]

    USEV: Universal speaker extraction with visual cue,

    Z. Pan, M. Ge, and H. Li, “USEV: Universal speaker extraction with visual cue,” IEEE/ACM Trans. Audio, Speech, Lang. Pro- cess., vol. 30, pp. 3032–3045, 2022

    2022

  22. [30]

    Real-time audio-visual end-to-end speech enhance- ment,

    Z. Zhu, H. Yang, M. Tang, Z. Yang, S. E. Eskimez, and H. Wang, “Real-time audio-visual end-to-end speech enhance- ment,” in Proc. ICASSP, 2023

    2023

  23. [31]

    BlazeFace: Sub-millisecond neural face detec- tion on mobile GPUs,

    V . Bazarevsky, Y . Kartynnik, A. Vakunov, K. Raveendran, and M. Grundmann, “BlazeFace: Sub-millisecond neural face detec- tion on mobile GPUs,” in CVPR Workshop on Computer Vision for Augmented and Virtual Reality, 2019

    2019

  24. [32]

    Xception: Deep learning with depthwise separable convolutions,

    F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” inProc. CVPR, 2017, pp. 1800–1807

    2017

  25. [33]

    PARIS: Pseudo-autoregressive siamese training for online speech separation,

    Z. Pan, G. Wichern, F. G. Germain, K. Saijo, and J. Le Roux, “PARIS: Pseudo-autoregressive siamese training for online speech separation,” in Proc. Interspeech, 2024

    2024

  26. [34]

    LRS3-TED: a large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “LRS3-TED: a large-scale dataset for visual speech recognition,” preprint arXiv:1809.00496, 2018

    2018 arXiv

  27. [35]

    Deep lip reading: A comparison of models and an online application,

    ——, “Deep lip reading: A comparison of models and an online application,” in Proc. Interspeech, 2018, pp. 3514–3518

    2018

  28. [36]

    MuSE: Multi-modal target speaker extraction with visual cues,

    Z. Pan, R. Tao, C. Xu, and H. Li, “MuSE: Multi-modal target speaker extraction with visual cues,” in Proc. ICASSP, 2021, pp. 6678–6682

    2021

  29. [37]

    SDR– half-baked or well done?

    J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR– half-baked or well done?” in Proc. ICASSP, 2019, pp. 626–630

    2019

  30. [38]

    A hybrid continuity loss to reduce over-suppression for time-domain target speaker extraction,

    Z. Pan, M. Ge, and H. Li, “A hybrid continuity loss to reduce over-suppression for time-domain target speaker extraction,” in Proc. Interspeech, 2022, pp. 1786–1790

    2022

  31. [39]

    ReVISE: Self-supervised speech resynthesis with visual input for universal and generalized speech regeneration,

    W.-N. Hsu, T. Remez, B. Shi, J. Donley, and Y . Adi, “ReVISE: Self-supervised speech resynthesis with visual input for universal and generalized speech regeneration,” in Proc. CVPR, 06 2023, pp. 18 795–18 805

    2023

  32. [40]

    PIA VE: A pose-invariant audio- visual speaker extraction network,

    Q. Liu, M. Ge, Z. Wu, and H. Li, “PIA VE: A pose-invariant audio- visual speaker extraction network,” in Proc. Interspeech, 2023

    2023

  33. [41]

    Audio- visual speech separation in noisy environments with a lightweight iterative model,

    H. Martel, J. Richter, K. Li, X. Hu, and T. Gerkmann, “Audio- visual speech separation in noisy environments with a lightweight iterative model,” in Proc. Interspeech, 2023

    2023

  34. [42]

    Adam, a method for stochastic optimiza- tion,

    D. P. Kingma and J. Ba, “Adam, a method for stochastic optimiza- tion,” in Proc. ICLR, vol. 1412, 2015

    2015

  35. [43]

    Performance mea- surement in blind audio source separation,

    E. Vincent, R. Gribonval, and C. F ´evotte, “Performance mea- surement in blind audio source separation,” IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 4, pp. 1462–1469, 2006

    2006

  36. [44]

    Per- ceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,

    A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Per- ceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” in Proc. ICASSP, 2001, pp. 749–752

    2001

  37. [45]

    A short- time objective intelligibility measure for time-frequency weighted noisy speech,

    C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “A short- time objective intelligibility measure for time-frequency weighted noisy speech,” in Proc. ICASSP, 2010, pp. 4214–4217

    2010

  38. [46]

    V oxCeleb2: Deep speaker recognition,

    J. S. Chung, A. Nagrani, and A. Zisserman, “V oxCeleb2: Deep speaker recognition,” Proc. Interspeech, pp. 1086–1090, 2018

    2018

  39. [47]

    TCD-TIMIT: An audio-visual corpus of continuous speech,

    N. Harte and E. Gillen, “TCD-TIMIT: An audio-visual corpus of continuous speech,” IEEE Transactions on Multimedia , vol. 17, no. 5, pp. 603–615, 2015

    2015

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.