Pith. sign in

REVIEW 4 major objections 5 minor 36 references

Training Strategies for Modality Dropout Resilient Multi-Modal Target Speaker Extraction

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Modality dropout training produces a multi-modal target speaker extraction system that is resilient to missing audio or video, less sensitive to normalization-layer choice, and able to use extracted speech as enrolment.

desk verdict Plausible but confounded: MDT's robustness may be an artifact of its zero-embedding inference protocol rather than the dropout training itself. read the letter →

arxiv 2507.06566 v1 pith:PN4YAKYF submitted 2025-07-09 eess.AS

Srikanth Korse , Mohamed Elminshawi , Emanuel A. P. Habets , Srikanth Raj Chetupalli This is my paper · ORCID
classification eess.AS
keywords multi-modaltargetspeakerextractionmodalitydropouttrainingstrategyaudio-visualspeechseparationcausalself-enrolmentnormalizationlayersdual-pathRNN
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a simple training-time schedule, randomly dropping one of the two auxiliary streams with probability 1/3 and replacing its embedding with zeros, produces a multi-modal target speaker extraction system that stays close to its uni-modal performance when audio or video is missing, without the modality dominance seen with standard or multi-task training. The authors show this holds for non-causal and causal configurations and across two normalization-layer choices, whereas standard and multi-task training degrade sharply in the audio-only and video-only conditions depending on the normalization layer. They also show the dropout-trained system can use speech extracted from an earlier segment as the enrolment signal, which matters when the target speaker has no clean enrolment utterance. If true, the result means the robustness problem in multi-modal speaker extraction can be solved at the training objective level rather than by architectural redesign.

What carries the argument

The central object is the attentive embedding combination of Eqs. (5)-(8), which forms the combined embedding $E_t = w_{a,t} E_a + w_{v,t} E_{v,t}$ by a cross-attention sharpened by $\gamma = 2$ between the mixture representation and each modality embedding, together with the MDT training distribution of Eq. (10). Under MDT, with probability 1/3 both modalities are used, with probability 1/3 only the video branch is used while the audio embedding is replaced by the zero vector $0_{N\times 1}$, and with probability 1/3 only the audio branch is used while the video embedding is replaced by $0_{N\times T_M}$. The all-zero substitution is what lets the embedding-extraction network for a dropped modality be bypassed at inference, and the training-time sampling of missing-modality cases is what the paper credits for the system's reduced sensitivity to normalization layers and its avoidance of modality dominance.

What would settle it

On a held-out test set with genuine modality absence, such as a video stream entirely removed rather than zeroed or an enrolment utterance replaced by silence, an MDT-trained model should maintain its audio-only and video-only SI-SDR improvements. If the model's AoTSE or VoTSE performance drops sharply or collapses when the modality is truly absent rather than zero-substituted, the central claim fails. A second concrete check is to inspect the attention weights $w_{a,t}$ and $w_{v,t}$ under zero-substitution: if they degenerate to a non-convex or saturated pattern that differs between zeroed and truly-absent inputs, the dropout simulation is not faithful.

Watch

Extended reading notes

Core claim

The central claim is that modality dropout training (MDT) yields a multi-modal target speaker extraction system with reduced sensitivity to architectural choices, notably the normalization layers, compared to standard training and multi-task training, in both non-causal and causal configurations. On two-speaker mixtures from LRS3, MDT-trained models show audio-only and video-only extraction performance close to the corresponding uni-modal systems, whereas standard and MTT-trained models exhibit large performance drops in those conditions and their rankings depend on whether global or layer normalization is used. The paper also reports that MDT-trained systems avoid modality dominance and, in a simulated self-enrolment scenario where the extracted speech of the previous segment is used as the enrolment, outperform the other strategies in both causal and non-causal configurations. The authors state that the MDT scheme is effective irrespective of the employed normalization layer.

Load-bearing premise

The claim rests on the assumption that replacing a missing modality's embedding with an all-zero vector during training faithfully mimics a genuinely absent modality at inference, so that the attentive convex combination of Eq. (8) remains a valid weighting when one branch is zero.

Editorial extensions

If this is right

  • Practitioners can build modality-robust MTSE by adding a one-third dropout schedule over the auxiliary streams, without changing the fusion architecture or adding extra loss terms.
  • The reduced sensitivity to normalization layers means the training strategy, not the choice of gLN, LN, or cLN, becomes the main lever for robustness to missing modalities.
  • The self-enrolment result implies MTSE systems can operate when the target speaker has no pre-recorded enrolment, using the model's own previous output as the enrolment signal.
  • Because dropped-modality embeddings are zeros, inference can skip the corresponding clue network, saving computation in audio-only or video-only conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper evaluates only two-speaker clean mixtures; an obvious testable extension is whether the MDT benefit persists with noisy mixtures, multiple interferers, or reverberation, since the conclusion section itself hypothesizes this but does not test it.
  • The use of zero embeddings as a dropout substitute assumes absence is equivalent to a zero vector in the cross-attention space; a natural follow-up is to compare zero-substitution against actual modality removal or learned 'missing' embeddings to see whether the robustness transfers.
  • The reported robustness might interact with the specific dual-path RNN backbone and attentive combination; testing MDT with other fusion methods, such as summation or concatenation, would delimit how general the claim is.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper compares three training strategies—standard training (ST), multi-task training (MTT), and modality dropout training (MDT)—for a multi-modal target speaker extraction (MTSE) system that uses audio enrolment and visual stream as auxiliary information. The authors report that MDT yields reduced sensitivity to the choice of normalization layer (gLN/LN or cLN/LN) in both non-causal and causal configurations, avoids modality dominance in audio-only and video-only inference conditions, and performs better under a simulated self-enrolment scenario. Experiments are conducted on two-speaker mixtures from LRS3, with SI-SDR improvement as the metric. The paper is an empirical comparison with clearly specified architecture and hyperparameters, but the evaluation protocol differs between training strategies and no significance tests are reported.

Significance. If the central claim holds, a simple training-time dropout schedule plus zero-embedding inference would be a practical recipe for making MTSE robust to missing modalities across architectural choices. The paper is useful in concretely identifying normalization-layer sensitivity as a key issue and in proposing a straightforward remedy. Its strengths include transparent specification of model and training details, a reasonable set of inference conditions (MTSE, AoTSE, V oTSE, MTSE-FD), and an interesting self-enrolment experiment. However, the lack of a matched inference protocol across compared strategies and the absence of statistical testing undermine the strength of the conclusions as stated.

major comments (4)
  1. [Section 5.3, Eq. (10)] The evaluation protocol is not matched across training strategies. For ST and MTT, missing modalities are simulated by zeroing the raw input signals (sv=0 or sa=0) and passing them through the full clue networks, whereas for MDT the missing modality is simulated by zeroing the embedding itself (Ev=0 or Ea=0) and bypassing the corresponding clue network. Because the MDT system is both trained and tested under the zero-embedding protocol, Table 1 does not isolate the effect of the training strategy from the effect of the inference protocol. I recommend a matched evaluation—for example, evaluating ST and MTT models with zeroed embeddings and evaluating the MDT model with zeroed raw inputs—or a clear justification for why the current protocol is the only meaningful one for each strategy.
  2. [Eqs. (5)-(8) and Eq. (10)] The assumption that replacing a dropped modality's embedding with an all-zero vector faithfully represents modality absence is not validated. Under the attentive combination in Eqs. (5)-(8), when one branch is zero the surviving embedding is multiplied by a softmax weight strictly less than 1 (e.g., w_a,t = sigmoid(...) when Ev=0), so the available modality is time-varying attenuated relative to a clean single-modality presentation. This could create a training-only or inference-only artifact rather than true robustness. Please analyze the learned gating weights under AoTSE/V oTSE conditions, or compare MDT against a 'hard selection' variant that sets the available modality weight to 1 when the other branch is dropped, to demonstrate that the zero-substitution is benign.
  3. [Table 1 and Figure 3] The central quantitative claims rely on means and standard deviations over test examples without significance tests or multiple random seeds. Some differences that support the conclusions are small in absolute terms (e.g., non-causal VoTSE: MTT LN 14.5 vs. MDT LN 13.9; causal VoTSE cLN: MTT 12.5 vs. MDT 12.3). To support claims such as 'MDT is less prone to modality dominance' and 'MDT is robust to the normalization layer choice,' please report paired significance tests (e.g., matched pairs across conditions or across normalization layers) or bootstrap confidence intervals, and clarify whether the standard deviations are across test examples or across model runs.
  4. [Section 6 and Section 4.2] The MTT baseline is altered after observing a failure: the paper states that the MTT strategy with weights proposed in [9] and gLN 'completely failed' for AoTSE, and therefore equal weights were used. This post hoc hyperparameter change should be justified with a sensitivity analysis. Without a small study of MTT loss weights, the reader cannot tell whether the reported MTT results are representative of the strategy or are a fortuitously favorable (or unfavorable) configuration.
minor comments (5)
  1. [Section 1] The text says MDT 'remains unexplored' in multi-modal TSE, but then cites prior uses of modality dropout in other domains [13-16]; please clarify the novelty as 'unexplored in the context of multi-modal TSE' to avoid the impression of a contradiction.
  2. [Table 1] Please state explicitly whether the reported standard deviations are computed over the test-set examples or over multiple training runs; this is important for interpreting the variability.
  3. [Section 5.2] The dataset creation is based on the recipe of [24]; please specify at least the number of distinct speaker identities in the training/test splits and the exact SIR sampling procedure, or state that this is fully provided in [24].
  4. [Figure 2] The caption and the surrounding text contain duplicated '3 s' labels ('3 s3 s'), which makes the timeline hard to read; please correct the formatting.
  5. [Section 7] The concluding hypothesis that conclusions hold for noisy mixtures is untested; please label it explicitly as speculation or add a small noisy-mixture experiment.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the MDT result is an empirical comparison, not a derivation that reduces to its inputs.

full rationale

The paper's central claim is empirical: models trained with modality dropout training (MDT) show reduced sensitivity to normalization layers and less modality dominance than standard or multi-task training. This is not a derivation, and no equation in the paper is defined in terms of the quantity it purports to predict. The MDT training objective in Eq. (10) is a dropout schedule over embeddings, and the inference protocol in Sec. 5.3 evaluates the system under exactly the missing-modality conditions used in training; this is consistency, not circularity. The only self-referential element is the dataset-creation recipe cited as [24], which is from the same group, but it is used only to generate two-speaker LRS3 mixtures and does not by construction force any of the reported SI-SDR improvements. Similarly, the modification of the MTT loss weights in Sec. 6 is an acknowledged adaptation of baseline [9], not a fitted parameter renamed as a prediction. The skeptic's concern about zero-embedding substitution attenuating the surviving modality is a validity or generalization critique of the missing-modality simulation, not a circularity: the MDT-trained model is optimized to work with zero embeddings, so evaluating it with zero embeddings is a fair within-system test, and that protocol does not make the normalization-layer robustness claim true by construction. No load-bearing step reduces to a self-citation or to the fitted training signal.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and introduces no new theoretical entities. The only invented mechanism is the all-zero embedding substitution, which is a training/inference scheme rather than a new physical or model entity; it is counted as an assumption above.

free parameters (4)
  • Modality dropout probabilities = 1/3 each for both-only, audio-only, video-only
    Chosen by hand; no ablation is reported, so the robustness result could depend on this specific schedule.
  • MTT loss weights = 1/3, 1/3, 1/3
    Original weights from [9] caused complete failure with gLN in AoTSE, so equal weights were adopted post hoc; the MTT comparison depends on this modification.
  • Sharpening factor gamma = 2
    Set as in [11]; affects the softmax weighting in Eq. (8) but is not a fitted constant for the main claim.
  • SIR range = -5 to 5 dB
    Data generation choice; the self-enrolment and robustness results are only demonstrated within this range.
assumptions (4)
  • domain assumption The input mixture contains exactly two speakers, cleanly mixed, with interference only in the audio; the video stream contains only the target speaker.
    Section 2 problem formulation states this directly; it excludes noisy, multi-speaker, or occluded-video conditions that real systems face.
  • ad hoc to paper Replacing a dropped modality's embedding with an all-zero vector is a faithful model of modality absence.
    Used in Eq. (10) for training and in Section 5.3 for inference; the paper does not validate this equivalence against other ways of masking, such as gating or learned missingness tokens.
  • domain assumption The pretrained visual front-end Fv from [21] provides reliable lip features for LRS3.
    Section 5.1 freezes this front-end; if lip features were unreliable, the video channel's contribution and the VoTSE conclusions would change.
  • domain assumption SI-SDR improvement is a sufficient metric for comparing these strategies.
    All conclusions are drawn from SI-SDR (Table 1); no intelligibility, subjective, or robustness-at-failure metrics are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training Strategies for Modality Dropout Resilient Multi-Modal Target Speaker Extraction." pith.science (2026). https://pith.science/paper/PN4YAKYF

@misc{pith2026250706566,
  author       = {Pith},
  title        = {Pith review of: Training Strategies for Modality Dropout Resilient Multi-Modal Target Speaker Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PN4YAKYF}},
  note         = {Machine review of arXiv:2507.06566}
}
read the original abstract

The primary goal of multi-modal TSE (MTSE) is to extract a target speaker from a speech mixture using complementary information from different modalities, such as audio enrolment and visual feeds corresponding to the target speaker. MTSE systems are expected to perform well even when one of the modalities is unavailable. In practice, the systems often suffer from modality dominance, where one of the modalities outweighs the others, thereby limiting robustness. Our study investigates training strategies and the effect of architectural choices, particularly the normalization layers, in yielding a robust MTSE system in both non-causal and causal configurations. In particular, we propose the use of modality dropout training (MDT) as a superior strategy to standard and multi-task training (MTT) strategies. Experiments conducted on two-speaker mixtures from the LRS3 dataset show the MDT strategy to be effective irrespective of the employed normalization layer. In contrast, the models trained with the standard and MTT strategies are susceptible to modality dominance, and their performance depends on the chosen normalization layer. Additionally, we demonstrate that the system trained with MDT strategy is robust to using extracted speech as the enrollment signal, highlighting its potential applicability in scenarios where the target speaker is not enrolled.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 33 canonical work pages

  1. [9]

    The cocktail-party problem revisited: Early processing and selection of multi-talker speech

    A. W. Bronkhorst, “The cocktail-party problem revisited: Early processing and selection of multi-talker speech.” At- tention, Perception and Psychophysics , vol. 77, no. 5, p. 1465–1487, 2015

    work page 2015

  2. [1]

    Humans use auxiliary information, such as spatial and visual cues as well as speaker familiarity, to selectively attend to auditory stimuli [1]

    INTRODUCTION The ability of humans to listen to a single speaker in an environ- ment with interfering acoustic sources is commonly referred to as the cocktail-party effect [1]. Humans use auxiliary information, such as spatial and visual cues as well as speaker familiarity, to selectively attend to auditory stimuli [1]. However, it is challenging for huma...

    work page

  3. [2]

    PROBLEM FORMULA TION Consider the scenario of an audio-visual recording of a single speaker of interest (i.e., the target speaker) and an interfering speaker. In this work, we assume the interference to be from a dif- arXiv:2507.06566v1 [eess.AS] 9 Jul 2025 Encoder (E) DNN1 Fusion DNN2 × Decoder (D) AudioClueNet Ea Audio Net Avg VideoClueNet Fv Video Net ...

    work page Pith review arXiv 2025

  4. [3]

    The architecture comprises an AudioClueNet module, a VideoClueNet module, an embedding combination module, and an extraction network

    MULTI-MODAL TARGET SPEAKER EXTRACTION Figure 1 shows a block diagram of the MTSE system used in this study. The architecture comprises an AudioClueNet module, a VideoClueNet module, an embedding combination module, and an extraction network. The extraction network uses ”masking in the learned time- feature domain” [18] strategy in which the input speech m...

    work page

  5. [4]

    3 was trained using three differ- ent training strategies to study their effect on the model’s robustness

    TRAINING STRA TEGIES The MTSE model described in Sec. 3 was trained using three differ- ent training strategies to study their effect on the model’s robustness. 4.1. Standard training (ST) In the standard training strategy, the model is trained with the scale- invariant source-to-distortion ratio (SI-SDR) [19] as a loss function by feeding the model with ...

    work page

  6. [5]

    Model description The basic building block of MTSE system under test is the dual- path recurrent neural network (DPRNN) proposed in [20]

    EXPERIMENTAL SETUP 5.1. Model description The basic building block of MTSE system under test is the dual- path recurrent neural network (DPRNN) proposed in [20]. The time- feature encoders E and Ea are realized by a 1D convolutional layer with 2 ms kernel size, 1 ms stride and 256 channels. The visual front-end Fv [21] is pre-trained and extracts a 512-di...

    work page

  7. [6]

    Our initial experimentation showed the models to be sensitive to the normalization layers used in the DNN architecture

    RESULTS We evaluated the uni-modal and multi-modal TSE systems in non-causal and causal configurations. Our initial experimentation showed the models to be sensitive to the normalization layers used in the DNN architecture. Hence, we study the choice of the normaliza- tion layer in addition to training strategies. In particular, global layer normalization...

    work page

  8. [7]

    CONCLUSION We compared a MDT strategy for modality dropout resilient MTSE system training to the existing strategies in non-causal and causal configurations. A comparison of these strategies, in combination with the choice of normalization layers used by the DNN, showed that the MDT strategy is effective in diverse practical scenarios while the existing s...

    work page

Show all 36 references
  1. [8]

    Usev: Universal speaker extraction with visual cue,

    Z. Pan, M. Ge, and H. Li, “Usev: Universal speaker extraction with visual cue,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 3032–3045, Sept. 2022

    2022

  2. [10]

    Single channel target speaker extraction and recognition with speaker beam,

    M. Delcroix, K. Zmolikova, K. Kinoshita, A. Ogawa, and T. Nakatani, “Single channel target speaker extraction and recognition with speaker beam,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Apr. 2018, pp. 5554–5558

    2018

  3. [11]

    Improving speaker discrimination of target speech extraction with time-domain speakerbeam,

    M. Delcroix, T. Ochiai, K. Zmolikova, K. Kinoshita, N. Tawara, T. Nakatani, and S. Araki, “Improving speaker discrimination of target speech extraction with time-domain speakerbeam,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), May 2020, pp. 691–695

    2020

  4. [12]

    X-TaSNet: Robust and accu- rate time-domain speaker extraction network,

    Z. Zhang, B. He, and Z. Zhang, “X-TaSNet: Robust and accu- rate time-domain speaker extraction network,” in Proc. Inter- speech Conf., Oct. 2020, pp. 1421–1425

    2020

  5. [13]

    SpEx: Multi-scale time domain speaker extraction network,

    C. Xu, W. Rao, E. S. Chng, and H. Li, “SpEx: Multi-scale time domain speaker extraction network,”IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 28, pp. 1370–1384, Apr. 2020

    2020

  6. [14]

    Time domain audio visual speech separation,

    J. Wu, Y . Xu, S.-X. Zhang, L.-W. Chen, M. Yu, L. Xie, and D. Yu, “Time domain audio visual speech separation,” inProc. IEEE Workshop on Automatic Speech Recognition and Under- standing, Dec. 2019, pp. 667–673

    2019

  7. [15]

    Muse: Multi-modal target speaker extraction with visual cues,

    Z. Pan, R. Tao, C. Xu, and H. Li, “Muse: Multi-modal target speaker extraction with visual cues,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP) , June 2021, pp. 6678–6682

    2021

  8. [16]

    A universally- deployable ASR frontend for joint acoustic echo cancellation, speech enhancement, and voice separation,

    T. R. O’Malley, A. Narayanan, and Q. Wang, “A universally- deployable ASR frontend for joint acoustic echo cancellation, speech enhancement, and voice separation,” in Proc. Inter- speech Conf., Sept. 2022, pp. 3829–3833

    2022

  9. [17]

    Multimodal SpeakerBeam: Single channel tar- get speech extraction with audio-visual speaker clues,

    T. Ochiai, M. Delcroix, K. Kinoshita, A. Ogawa, and T. Nakatani, “Multimodal SpeakerBeam: Single channel tar- get speech extraction with audio-visual speaker clues,” inProc. Interspeech Conf., Sept. 2019, pp. 2718–2722

    2019

  10. [18]

    My lips are con- cealed: Audio-visual speech enhancement through obstruc- tions,

    T. Afouras, J. S. Chung, and A. Zisserman, “My lips are con- cealed: Audio-visual speech enhancement through obstruc- tions,” in Proc. Interspeech Conf., Sept. 2019, pp. 4295–4299

    2019

  11. [19]

    Multimodal attention fusion for target speaker extraction,

    H. Sato, T. Ochiai, K. Kinoshita, M. Delcroix, T. Nakatani, and S. Araki, “Multimodal attention fusion for target speaker extraction,” in IEEE Spoken Language Technology Workshop (SLT), Jan. 2021, pp. 778–784

    2021

  12. [20]

    An overview of deep-learning-based audio- visual speech enhancement and separation,

    D. Michelsanti, Z.-H. Tan, S.-X. Zhang, Y . Xu, M. Yu, D. Yu, and J. Jensen, “An overview of deep-learning-based audio- visual speech enhancement and separation,”IEEE/ACM Trans. Audio, Speech, Lang. Process. , vol. 29, pp. 1368–1396, Mar. 2021

    2021

  13. [21]

    Moddrop: Adaptive multi-modal gesture recognition,

    N. Neverova, C. Wolf, G. Taylor, and F. Nebout, “Moddrop: Adaptive multi-modal gesture recognition,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 38, no. 08, pp. 1692–1706, Aug. 2016

    2016

  14. [22]

    Modality dropout for im- proved performance-driven talking faces,

    A. Hussen Abdelaziz, B.-J. Theobald, P. Dixon, R. Knothe, N. Apostoloff, and S. Kajareker, “Modality dropout for im- proved performance-driven talking faces,” in Proc. Intl. Conf. on Multimodal Interaction (ICMI) , Oct. 2020, p. 378–386

    2020

  15. [23]

    Learnable irrele- vant modality dropout for multimodal action recognition on modality-specific annotated videos,

    S. Alfasly, J. Lu, C. Xu, and Y . Zou, “Learnable irrele- vant modality dropout for multimodal action recognition on modality-specific annotated videos,” in Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , June 2022, pp. 20 176–20 185

    2022

  16. [24]

    New insights on target speaker extraction,

    M. Elminshawi, W. Mack, S. Chakrabarty, and E. A. P. Ha- bets, “New insights on target speaker extraction,” Feb. 2022, arXiv:2202.00733

    2022 arXiv

  17. [25]

    Multi- stage speaker extraction with utterance and frame-level refer- ence signals,

    M. Ge, C. Xu, L. Wang, E. S. Chng, J. Dang, and H. Li, “Multi- stage speaker extraction with utterance and frame-level refer- ence signals,” inICASSP 2021 - 2021 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 6109–6113

    2021

  18. [26]

    TasNet: Time-domain audio sep- aration network for real-time, single-channel speech separa- tion,

    Y . Luo and N. Mesgarani, “TasNet: Time-domain audio sep- aration network for real-time, single-channel speech separa- tion,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Sig- nal Processing (ICASSP), Apr. 2018, pp. 696–700

    2018

  19. [27]

    SDR– half-baked or well done?

    J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR– half-baked or well done?” in Proc. IEEE Intl. Conf. on Acous- tics, Speech and Signal Processing (ICASSP) , May 2019, pp. 626–630

    2019

  20. [28]

    Dual-path rnn: Effi- cient long sequence modeling for time-domain single-channel speech separation,

    Y . Luo, Z. Chen, and T. Yoshioka, “Dual-path rnn: Effi- cient long sequence modeling for time-domain single-channel speech separation,” in Proc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP) , May 2020, pp. 46– 50

    2020

  21. [29]

    Combining residual net- works with lstms for lipreading,

    T. Stafylakis and G. Tzimiropoulos, “Combining residual net- works with lstms for lipreading,” in Proc. Interspeech Conf. , Aug. 2017, pp. 3652–3656

    2017

  22. [30]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) , June 2016, pp. 770– 778

    2016

  23. [31]

    LRS3-TED: A large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “LRS3-TED: A large-scale dataset for visual speech recognition,” Oct. 2018, arXiv:1809.00496

    2018 arXiv

  24. [33]

    Adam: A method for stochastic op- timization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic op- timization,” in Proc. Intl. Conf. on Learn. Repr . (ICLR) , May 2015, pp. 1–15

    2015

  25. [34]

    Wavesplit: End-to-end speech separation by speaker clustering,

    N. Zeghidour and D. Grangier, “Wavesplit: End-to-end speech separation by speaker clustering,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 29, pp. 2840–2849, July 2021

    2021

  26. [35]

    Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation,

    Y . Luo and N. Mesgarani, “Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation,” IEEE/ACM Trans. Audio, Speech, Lang. Process. , vol. 27, no. 8, pp. 1256–1266, Aug. 2019

    2019

  27. [36]

    Layer normalization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” July 2016, arXiv:1607.06450

    2016 arXiv

  28. [128]

    The inter- and intra-chunk RNNs are realized in the non-causal configuration using bi-directional long short-term mem- ory (LSTM)

    A DPRNN layer consists of a cascade of inter-chunk and intra- chunk RNNs. The inter- and intra-chunk RNNs are realized in the non-causal configuration using bi-directional long short-term mem- ory (LSTM). However, for the causal configuration, inter-RNN uses a uni-directional ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.