{"id":"a4d260f3-e6c5-49f2-8592-1d144627b4d2","arxiv_id":"2507.06566","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Modality dropout training makes audio-visual target speaker extraction robust when audio or video is missing, across normalization layers and in causal configurations.","lead":"This paper tests a training trick, called modality dropout training, for multi-modal target speaker extraction, where a system pulls out one speaker's voice using both an audio sample and video of the speaker. It reports that the trick keeps the system working when one input is missing, and that it works with causal (low-latency) models too.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MDT's zero-embedding dropout does not truly 'drop' a modality: with the softmax in Eq. (8), the surviving embedding is attenuated by a learned sigmoid factor, so the reported robustness may be an artifact of this missing-modality representation rather than of the training strategy itself.","rationale":"The paper's central claim is that MDT reduces sensitivity to architectural choices and avoids modality dominance compared to standard and multi-task training. The table does show small norm-related differences for MDT and large ones for ST/MTT in several conditions, so the claim is plausible. However, the mechanism by which MDT simulates missing modalities is not a clean dropout: because the softmax in Eq. (8) always assigns some weight to the zero branch, the available branch is attenuated. This means the MDT model has never been trained or evaluated with a full-scale single-modality embedding, so its AoTSE/VoTSE behavior may be an artifact of the zero-embedding construction rather than of modality dropout training per se. A second confound is that the missing-modality implementation at inference differs across training strategies (zero raw input for MTT, zero embedding for MDT), so the comparison does not isolate the training loss. The proposed masked-attention retraining is a direct and inexpensive check: if the same robustness appears when the surviving branch is not attenuated, the concern is resolved; if not, the conclusion needs qualification. The reader identified the zero-embedding substitution as the weakest assumption; my attack sharpens it by showing the specific way the softmax distorts the surviving embedding and by noting the cross-strategy implementation confound. No code or data are available, and no significance testing is reported, but the main unresolved point is the faithfulness of the missing-modality simulation. The conditional verdict remains appropriate, so no change to the reader's verdict is recommended.","tokens_in":8909,"tokens_out":11693,"duration_ms":119372,"concrete_test":"Retrain the MDT models with the same dropout schedule but replace Eq. (8) with a masked softmax: when a modality is dropped, set its logit to -inf before normalization, so the surviving branch is used at full scale. Compare AoTSE/VoTSE SI-SDR improvement and the gLN-vs-LN (or cLN-vs-LN) differences against Table 1. If the norm-sensitivity gap persists, zero-embedding attenuation is not the driver; if it disappears or reverses, the reported MDT robustness is tied to the zero-embedding representation and the 'training strategy' claim must be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Eq. (10), MDT replaces a dropped branch with an all-zero embedding. But the attentive combination f in Eqs. (5)-(8) still runs softmax over both branches. Because W Ht + b is common to ea,t and ev,t, when one branch is zero the available branch's weight is w_avail = sigmoid(Delta) with Delta = w^T[tanh(W Ht + V E_avail + b) - tanh(W Ht + b)]. Hence the surviving embedding is scaled down by a factor strictly less than 1 unless Delta goes to infinity. The model is trained to work with an attenuated single-modality embedding, not with a clean one. This is not equivalent to actual modality absence, where the available modality should be usable at full strength. The comparison is also confounded: for MTT-trained systems, Sec. 5.3 evaluates missing modalities by zeroing raw inputs (sv=0 or sa=0), whereas for MDT-trained systems it bypasses the clue net and zeroes the embedding (Ev=0 or Ea=0). The central claim that MDT itself, as a training-time dropout schedule, reduces sensitivity to normalization layers is therefore not cleanly isolated; the observed robustness could be a property of the zero-embedding inference protocol rather than of the dropout loss. This is the load-bearing assumption behind the main claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares three training strategies—standard training (ST), multi-task training (MTT), and modality dropout training (MDT)—for a multi-modal target speaker extraction (MTSE) system that uses audio enrolment and visual stream as auxiliary information. The authors report that MDT yields reduced sensitivity to the choice of normalization layer (gLN/LN or cLN/LN) in both non-causal and causal configurations, avoids modality dominance in audio-only and video-only inference conditions, and performs better under a simulated self-enrolment scenario. Experiments are conducted on two-speaker mixtures from LRS3, with SI-SDR improvement as the metric. The paper is an empirical comparison with clearly specified architecture and hyperparameters, but the evaluation protocol differs between training strategies and no significance tests are reported.","tokens_in":9202,"tokens_out":5792,"duration_ms":68332,"significance":"If the central claim holds, a simple training-time dropout schedule plus zero-embedding inference would be a practical recipe for making MTSE robust to missing modalities across architectural choices. The paper is useful in concretely identifying normalization-layer sensitivity as a key issue and in proposing a straightforward remedy. Its strengths include transparent specification of model and training details, a reasonable set of inference conditions (MTSE, AoTSE, V oTSE, MTSE-FD), and an interesting self-enrolment experiment. However, the lack of a matched inference protocol across compared strategies and the absence of statistical testing undermine the strength of the conclusions as stated.","major_comments":[{"comment":"The evaluation protocol is not matched across training strategies. For ST and MTT, missing modalities are simulated by zeroing the raw input signals (sv=0 or sa=0) and passing them through the full clue networks, whereas for MDT the missing modality is simulated by zeroing the embedding itself (Ev=0 or Ea=0) and bypassing the corresponding clue network. Because the MDT system is both trained and tested under the zero-embedding protocol, Table 1 does not isolate the effect of the training strategy from the effect of the inference protocol. I recommend a matched evaluation—for example, evaluating ST and MTT models with zeroed embeddings and evaluating the MDT model with zeroed raw inputs—or a clear justification for why the current protocol is the only meaningful one for each strategy.","section":"Section 5.3, Eq. (10)"},{"comment":"The assumption that replacing a dropped modality's embedding with an all-zero vector faithfully represents modality absence is not validated. Under the attentive combination in Eqs. (5)-(8), when one branch is zero the surviving embedding is multiplied by a softmax weight strictly less than 1 (e.g., w_a,t = sigmoid(...) when Ev=0), so the available modality is time-varying attenuated relative to a clean single-modality presentation. This could create a training-only or inference-only artifact rather than true robustness. Please analyze the learned gating weights under AoTSE/V oTSE conditions, or compare MDT against a 'hard selection' variant that sets the available modality weight to 1 when the other branch is dropped, to demonstrate that the zero-substitution is benign.","section":"Eqs. (5)-(8) and Eq. (10)"},{"comment":"The central quantitative claims rely on means and standard deviations over test examples without significance tests or multiple random seeds. Some differences that support the conclusions are small in absolute terms (e.g., non-causal VoTSE: MTT LN 14.5 vs. MDT LN 13.9; causal VoTSE cLN: MTT 12.5 vs. MDT 12.3). To support claims such as 'MDT is less prone to modality dominance' and 'MDT is robust to the normalization layer choice,' please report paired significance tests (e.g., matched pairs across conditions or across normalization layers) or bootstrap confidence intervals, and clarify whether the standard deviations are across test examples or across model runs.","section":"Table 1 and Figure 3"},{"comment":"The MTT baseline is altered after observing a failure: the paper states that the MTT strategy with weights proposed in [9] and gLN 'completely failed' for AoTSE, and therefore equal weights were used. This post hoc hyperparameter change should be justified with a sensitivity analysis. Without a small study of MTT loss weights, the reader cannot tell whether the reported MTT results are representative of the strategy or are a fortuitously favorable (or unfavorable) configuration.","section":"Section 6 and Section 4.2"}],"minor_comments":[{"comment":"The text says MDT 'remains unexplored' in multi-modal TSE, but then cites prior uses of modality dropout in other domains [13-16]; please clarify the novelty as 'unexplored in the context of multi-modal TSE' to avoid the impression of a contradiction.","section":"Section 1"},{"comment":"Please state explicitly whether the reported standard deviations are computed over the test-set examples or over multiple training runs; this is important for interpreting the variability.","section":"Table 1"},{"comment":"The dataset creation is based on the recipe of [24]; please specify at least the number of distinct speaker identities in the training/test splits and the exact SIR sampling procedure, or state that this is fully provided in [24].","section":"Section 5.2"},{"comment":"The caption and the surrounding text contain duplicated '3 s' labels ('3 s3 s'), which makes the timeline hard to read; please correct the formatting.","section":"Figure 2"},{"comment":"The concluding hypothesis that conclusions hold for noisy mixtures is untested; please label it explicitly as speculation or add a small noisy-mixture experiment.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a relevant problem and the MDT idea is simple and potentially useful. However, the core comparison is currently confounded by differing inference protocols, and the absence of significance testing and multiple seeds weakens the empirical claims. I would encourage the editor to request a matched-protocol experiment and a sensitivity analysis for the MTT baseline before considering publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is a decent empirical study of modality dropout training (MDT) for audio-visual target speaker extraction, and the headline result—MDT makes the system robust to normalization-layer choice in causal and non-causal settings—is plausible. But the comparison is contaminated by a mismatch in how missing modalities are simulated for MDT versus the baselines.\n\nWhat is new: MDT itself is not new (refs 13–16), but this is the first systematic application to MTSE, and the paper adds a worthwhile comparison across gLN/LN/cLN, causal/non-causal, and a self-enrolment experiment. The authors report clear hyperparameters, use a sensible dual-path architecture, and Table 1 does show the MDT rows are stable across norms while ST and MTT are not. That stability is the paper's real contribution.\n\nThe biggest soft spot is the inference-protocol confound. For MTT and ST, missing modalities are simulated by zeroing raw inputs (sv=0 or sa=0), so the clue nets still run and produce some arbitrary embedding. For MDT, they zero the embedding itself (Sec. 5.3). This means MDT models are trained and evaluated with a clean 'this modality is absent' signal, while the baselines are not. The observed robustness may owe more to the zero-embedding protocol than to the dropout loss. The stress-test note is right about the attention math: when one branch is zero, the softmax still attenuates the surviving branch by a sigmoid factor, so the model never learns to use the available modality at full strength. It learns to cope with an attenuated version. That is a legitimate system design, but it should be compared fairly against the baselines—e.g., evaluate MTT- and ST-trained models with zeroed embeddings too, and MDT with zeroed raw inputs.\n\nOther issues are secondary: no significance tests, one dataset, clean mixtures only, no code/data. The MTT baseline was adjusted post hoc (equal weights because the original weights failed with gLN); the authors disclose this, but it weakens the 'best baseline' claim.\n\nOverall, the paper deserves a serious peer review, but the central claim needs one clean ablation before I'd trust the generalization. The fix is straightforward, so this should be revisable.\n\nRecommendation: send to peer review, with a request for the ablation and significance testing.","headline":"Plausible but confounded: MDT's robustness may be an artifact of its zero-embedding inference protocol rather than the dropout training itself.","tokens_in":9748,"tokens_out":3862,"would_cite":false,"duration_ms":43989,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Modality dropout training produces a multi-modal target speaker extraction system that is resilient to missing audio or video, less sensitive to normalization-layer choice, and able to use extracted speech as enrolment.","keywords":["multi-modal target speaker extraction","modality dropout","training strategy","audio-visual speech separation","causal speaker extraction","self-enrolment","normalization layers","dual-path RNN"],"falsifier":"On a held-out test set with genuine modality absence, such as a video stream entirely removed rather than zeroed or an enrolment utterance replaced by silence, an MDT-trained model should maintain its audio-only and video-only SI-SDR improvements. If the model's AoTSE or VoTSE performance drops sharply or collapses when the modality is truly absent rather than zero-substituted, the central claim fails. A second concrete check is to inspect the attention weights $w_{a,t}$ and $w_{v,t}$ under zero-substitution: if they degenerate to a non-convex or saturated pattern that differs between zeroed and truly-absent inputs, the dropout simulation is not faithful.","tokens_in":8707,"feed_emoji":"🎧","tokens_out":4230,"duration_ms":40126,"temperature":0.7,"pith_summary":"This paper claims that a simple training-time schedule, randomly dropping one of the two auxiliary streams with probability 1/3 and replacing its embedding with zeros, produces a multi-modal target speaker extraction system that stays close to its uni-modal performance when audio or video is missing, without the modality dominance seen with standard or multi-task training. The authors show this holds for non-causal and causal configurations and across two normalization-layer choices, whereas standard and multi-task training degrade sharply in the audio-only and video-only conditions depending on the normalization layer. They also show the dropout-trained system can use speech extracted from an earlier segment as the enrolment signal, which matters when the target speaker has no clean enrolment utterance. If true, the result means the robustness problem in multi-modal speaker extraction can be solved at the training objective level rather than by architectural redesign.","feed_headline":"Dropout training saves speaker extraction when a modality goes missing","feed_subtitle":"A one-third chance of dropping audio or video during training keeps the model close to uni-modal performance when one cue is absent.","key_machinery":"The central object is the attentive embedding combination of Eqs. (5)-(8), which forms the combined embedding $E_t = w_{a,t} E_a + w_{v,t} E_{v,t}$ by a cross-attention sharpened by $\\gamma = 2$ between the mixture representation and each modality embedding, together with the MDT training distribution of Eq. (10). Under MDT, with probability 1/3 both modalities are used, with probability 1/3 only the video branch is used while the audio embedding is replaced by the zero vector $0_{N\\times 1}$, and with probability 1/3 only the audio branch is used while the video embedding is replaced by $0_{N\\times T_M}$. The all-zero substitution is what lets the embedding-extraction network for a dropped modality be bypassed at inference, and the training-time sampling of missing-modality cases is what the paper credits for the system's reduced sensitivity to normalization layers and its avoidance of modality dominance.","core_discovery":"The central claim is that modality dropout training (MDT) yields a multi-modal target speaker extraction system with reduced sensitivity to architectural choices, notably the normalization layers, compared to standard training and multi-task training, in both non-causal and causal configurations. On two-speaker mixtures from LRS3, MDT-trained models show audio-only and video-only extraction performance close to the corresponding uni-modal systems, whereas standard and MTT-trained models exhibit large performance drops in those conditions and their rankings depend on whether global or layer normalization is used. The paper also reports that MDT-trained systems avoid modality dominance and, in a simulated self-enrolment scenario where the extracted speech of the previous segment is used as the enrolment, outperform the other strategies in both causal and non-causal configurations. The authors state that the MDT scheme is effective irrespective of the employed normalization layer.","pith_inferences":["The paper evaluates only two-speaker clean mixtures; an obvious testable extension is whether the MDT benefit persists with noisy mixtures, multiple interferers, or reverberation, since the conclusion section itself hypothesizes this but does not test it.","The use of zero embeddings as a dropout substitute assumes absence is equivalent to a zero vector in the cross-attention space; a natural follow-up is to compare zero-substitution against actual modality removal or learned 'missing' embeddings to see whether the robustness transfers.","The reported robustness might interact with the specific dual-path RNN backbone and attentive combination; testing MDT with other fusion methods, such as summation or concatenation, would delimit how general the claim is."],"forward_implications":["Practitioners can build modality-robust MTSE by adding a one-third dropout schedule over the auxiliary streams, without changing the fusion architecture or adding extra loss terms.","The reduced sensitivity to normalization layers means the training strategy, not the choice of gLN, LN, or cLN, becomes the main lever for robustness to missing modalities.","The self-enrolment result implies MTSE systems can operate when the target speaker has no pre-recorded enrolment, using the model's own previous output as the enrolment signal.","Because dropped-modality embeddings are zeros, inference can skip the corresponding clue network, saving computation in audio-only or video-only conditions."],"supporting_citations":[{"why":"Supplies the multi-task training baseline and the element-wise fusion of mixture representations with combined embeddings.","marker":"[9]"},{"why":"Provides the attentive embedding-combination scheme and the sharpening factor gamma used in the cross-attention weighting.","marker":"[11]"},{"why":"Introduces modality dropout in another domain and serves as the direct inspiration for the MDT training strategy.","marker":"[13]"},{"why":"Defines the SI-SDR loss function used to train all three strategies and to evaluate extraction performance.","marker":"[19]"},{"why":"Supplies the dual-path recurrent neural network backbone that forms the extraction network and the clue networks.","marker":"[20]"},{"why":"Provides the LRS3 audio-visual dataset from which the two-speaker mixtures are generated.","marker":"[23]"},{"why":"Defines the mixture creation procedure, including random speaker pairing and signal-to-interference ratios from -5 to 5 dB.","marker":"[24]"},{"why":"Defines the global and cumulative layer normalization layers whose choice is varied in the normalization-sensitivity experiments.","marker":"[27]"}],"fun_headline_variants":["Random dropout during training boosts multimodal speaker extraction","Modality dropout training beats standard and multi-task methods","Dropout training keeps speaker extraction robust to missing cues","Train with random modality dropout for resilient speaker extraction","Modality dropout training reduces sensitivity to normalization choices"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that replacing a missing modality's embedding with an all-zero vector during training faithfully mimics a genuinely absent modality at inference, so that the attentive convex combination of Eq. (8) remains a valid weighting when one branch is zero.","fun_headline_variants_meta":{"raw":{"variants":["Random dropout during training boosts multimodal speaker extraction","Modality dropout training beats standard and multi-task methods","Dropout training keeps speaker extraction robust to missing cues","Train with random modality dropout for resilient speaker extraction","Modality dropout training reduces sensitivity to normalization choices"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1206,"prompt_tokens":930,"completion_tokens":276,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":204}},"tokens_in":546,"tokens_out":276,"duration_ms":3818,"temperature":1.0,"reasoning_tokens":204,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:59:39.807038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out test set with genuine modality absence, such as a video stream entirely removed rather than zeroed or an enrolment utterance replaced by silence, an MDT-trained model should maintain its audio-only and video-only SI-SDR improvements. If the model's AoTSE or VoTSE performance drops sharply or collapses when the modality is truly absent rather than zero-substituted, the central claim fails. A second concrete check is to inspect the attention weights $w_{a,t}$ and $w_{v,t}$ under zero-substitution: if they degenerate to a non-convex or saturated pattern that differs between zeroed and truly-absent inputs, the dropout simulation is not faithful.","supporting_citations":[{"cited_title":"The cocktail-party problem revisited: Early processing and selection of multi-talker speech","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-task training baseline and the element-wise fusion of mixture representations with combined embeddings."},{"cited_title":"Improving speaker discrimination of target speech extraction with time-domain speakerbeam,","cited_arxiv_id":null,"evidence_quote":"Provides the attentive embedding-combination scheme and the sharpening factor gamma used in the cross-attention weighting."},{"cited_title":"SpEx: Multi-scale time domain speaker extraction network,","cited_arxiv_id":null,"evidence_quote":"Introduces modality dropout in another domain and serves as the direct inspiration for the MDT training strategy."},{"cited_title":"Multimodal attention fusion for target speaker extraction,","cited_arxiv_id":null,"evidence_quote":"Defines the SI-SDR loss function used to train all three strategies and to evaluate extraction performance."},{"cited_title":"An overview of deep-learning-based audio- visual speech enhancement and separation,","cited_arxiv_id":null,"evidence_quote":"Supplies the dual-path recurrent neural network backbone that forms the extraction network and the clue networks."},{"cited_title":"Learnable irrele- vant modality dropout for multimodal action recognition on modality-specific annotated videos,","cited_arxiv_id":null,"evidence_quote":"Provides the LRS3 audio-visual dataset from which the two-speaker mixtures are generated."},{"cited_title":"SDR– half-baked or well done?","cited_arxiv_id":null,"evidence_quote":"Defines the global and cumulative layer normalization layers whose choice is varied in the normalization-sensitivity experiments."}],"review_version":1}