{"id":"a8086ac3-78e2-40ef-a134-46e796b6b837","arxiv_id":"2505.12288","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A speaker-embedding-free enhancement model is extended to do both conventional denoising and target-speaker extraction with a zero-enrollment trick, plus a consistency loss that pairs two enrollment utterances of the same speaker to improve robustness.","lead":"This paper proposes two training methods that make one speech enhancement model handle both plain denoising and extracting a chosen speaker's voice, using either no reference or a voice sample as the guide. The second method trains the model with two different voice samples of the same speaker and forces its outputs to match, which the authors say makes the model focus on speaker identity and ignore emotion or wording.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DSEF-PNet gains may stem from two-enrollment data augmentation rather than the HEIT consistency loss; missing λ=0 ablation breaks the disentanglement attribution.","rationale":"The paper's headline contribution is that a single SEF-PNet architecture can handle both SE and PSE (USEF-PNet) and that an unsupervised consistency constraint (HEIT) disentangles enrollment speech to improve robustness (DSEF-PNet). The strongest claim combines both: a single lightweight model that is also robust to enrollment variation. The DSEF-PNet result is the more novel and causally loaded contribution because it introduces a new training objective (Eq. 5) and claims a specific mechanism (disentanglement). However, the experimental design does not isolate the mechanism. DSEF-PNet training uses two enrollment utterances per mixture and a weighted combination of two SISDR losses plus the HEIT term. Compared to SEF-PNet, which uses one enrollment per mixture, this is a two-fold increase in enrollment diversity per optimization step. A plausible alternative explanation for the gains in Table III (e.g., SISDR 13.57 vs 13.00 on 2-speaker) is that the model benefits from seeing more enrollment variations, not from the consistency constraint. Without a λ=0 ablation, the paper cannot rule out this trivial explanation. This is more load-bearing than the reader's stated concern about identity-relevant differences in enrollments: even if the consistency assumption holds, the consistency loss might contribute nothing. The missing λ report compounds the issue, preventing replication. Therefore, the central claim of unsupervised disentanglement is currently unsubstantiated. A simple control experiment—training with the same two-enrollment setup but λ=0—would settle the question. If the λ=0 baseline matches DSEF-PNet, the 'disentanglement' contribution reduces to an enrollment augmentation trick; if it does not, the HEIT mechanism gains support. The reader's CONDITIONAL verdict remains appropriate, conditioned on this ablation, so I leave the verdict unchanged.","tokens_in":19226,"tokens_out":12899,"duration_ms":127341,"concrete_test":"Train DSEF-PNet with the same architecture and data but set λ=0 in Eq. (5), using only the two SISDR losses without the HEIT consistency term, on the Libri2Mix '2-speaker' condition. If this λ=0 baseline matches DSEF-PNet's SISDR/PESQ/STOI within ~0.1 dB, the HEIT loss is not the driver and the improvement is attributable to enrollment augmentation; if it falls back to SEF-PNet levels (13.00 SISDR), the consistency constraint is essential. Additionally, report the chosen λ value used for the main results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DSEF-PNet's HEIT loss (Eq. 4) 'implicitly disentangles' speaker identity from enrollment content rests on attributing the performance gains in Table III and Table V to the consistency constraint. But the DSEF-PNet training objective (Eq. 5) differs from SEF-PNet in two ways simultaneously: (1) each mixture is paired with two distinct enrollment utterances per optimization step, and (2) an L1 consistency term λ·L_HEIT is added. This creates a confound: the two-enrollment pairing acts as enrollment data augmentation and an enlarged effective batch, which by itself could improve SISDR and PESQ without any consistency constraint. The paper does not ablate the HEIT term (e.g., λ=0) to show that the consistency loss, rather than the doubled enrollment sampling, causes the observed gains. Additionally, λ is never reported, so the strength of the constraint is unknown and the results are not reproducible. Consequently, the paper's mechanism attribution—unsupervised disentanglement—is not empirically supported. If a λ=0 baseline reproduces the 2-speaker improvements (13.57 vs 13.00), the HEIT loss is superfluous and the disentanglement narrative collapses.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the authors' previous SEF-PNet speaker-encoder-free framework in two directions. USEF-PNet unifies conventional speech enhancement (SE) and personalized speech enhancement (PSE) by training the same architecture on mini-batches that contain both PSE pairs (mixture, enrollment) and SE samples with a zero-valued dummy enrollment, so that a single model can switch tasks at inference depending on whether enrollment is available. DSEF-PNet introduces Heterogeneous Enrollment Invariant Training (HEIT): for each mixture, two different enrollment utterances of the same target speaker are used in parallel, and the two enhanced outputs are forced to match with an L1 consistency loss in addition to the usual SISDR loss, with the goal of removing enrollment-content variation while retaining speaker identity. A Long-Short Enrollment Pairing (LSEP) variant is also explored. Experiments on Libri2Mix and VoiceBank-DEMAND report that USEF-PNet matches or slightly surpasses task-specific baselines, and that DSEF-PNet improves PSE metrics, most notably SISDR from 13.00 to 13.57 dB in the full Libri2Mix 2-speaker condition, while adding no extra parameters or inference overhead.","tokens_in":19413,"tokens_out":8110,"duration_ms":78002,"significance":"If the claims are substantiated, the paper offers an appealingly simple way to unify SE and PSE in one lightweight model, and a training-only mechanism for making PSE robust to enrollment variability—both valuable for resource-constrained deployment. The manuscript has concrete strengths: the USEF-PNet idea is clean and directly tested; the authors honestly report that the LSEP strategy does not improve over random enrollment durations; and the promise of released code and checkpoints supports reproducibility. The main weakness is that the central mechanism attribution for DSEF-PNet, namely unsupervised disentanglement of speaker identity, is not actually demonstrated. The observed gains could stem from the two-enrollment sampling acting as data augmentation rather than from the HEIT consistency loss, and no ablation or direct diagnostic separates these explanations. The significance of the DSEF-PNet contribution therefore currently rests on an under-supported inference, although the underlying idea is plausible and testable.","major_comments":[{"comment":"The DSEF-PNet training procedure changes two factors at once relative to SEF-PNet: each mixture is now paired with two distinct enrollment utterances per optimization step, and the L1 HEIT consistency term is added to the loss. The reported gains (e.g., 13.00 to 13.57 dB SISDR in the 2-speaker condition) could therefore be caused by the two-enrollment sampling acting as data augmentation or by the enlarged effective batch, without any contribution from the consistency constraint. A necessary control is a lambda=0 ablation that still uses two enrollments per step; without it, the gains cannot be attributed to HEIT. In addition, the value of lambda in Eq. (5) is never reported, so the strength of the constraint is unknown and the experiments are not reproducible.","section":"Section IV-B, Eqs. (3)-(5), Table III"},{"comment":"The claim that HEIT 'implicitly disentangles' speaker identity from irrelevant enrollment content is not directly tested. The paper reports only downstream enhancement metrics; there is no diagnostic showing that identity-relevant cues are preserved or that emotion, content, or noise are suppressed. Since the consistency loss could equally discard useful identity information when the two enrollments differ in identity-relevant ways, the use of the term 'disentanglement' is not supported by the presented evidence. An experiment that varies emotion or content in the enrollment while holding identity fixed, or an analysis of the internal representations, would be needed to support the mechanism claim.","section":"Section IV-B and Section VI-B"},{"comment":"The performance improvements attributed to DSEF-PNet are small (the largest SISDR gain is 0.57 dB, and several metrics are unchanged or improve by 0.03 or less), yet no error bars, number of random seeds, or significance tests are reported. The statements that DSEF-PNet 'consistently outperforms' SEF-PNet and yields 'significant' improvements are therefore not statistically substantiated, and the reader cannot determine whether the differences are within run-to-run variability.","section":"Tables III, IV, V and Section VI-B"},{"comment":"The SEF-PNet SE baseline on VoiceBank-DEMAND is underspecified. For the Table I experiments, the text states that the ISA module is removed from SEF-PNet to create a purely SE-focused variant, but no equivalent statement appears for the Table II SEF-PNet row. If the ISA module was retained and fed a zero or dummy enrollment, the comparison with USEF-PNet is not on equal footing, and the claimed SE improvement (19.21 to 19.69 dB SISDR) may reflect the baseline configuration rather than the unified training strategy.","section":"Section VI-A2, Table II"},{"comment":"The backbone architecture description is internally inconsistent. Section III states that the number of Encoder Blocks was reduced from 7 to 6 compared with sDPCCN, and describes the encoder as 'an ISA adaptation module followed by 6 Encoder Blocks,' while Section V-B states that the SEF-PNet architecture 'consists of 7 Encoder Blocks, 7 corresponding Decoder Blocks.' Since every experiment in the paper builds on this backbone, the contradiction prevents exact reproduction and should be resolved.","section":"Section III vs. Section V-B"}],"minor_comments":[{"comment":"The STOI values in Table II are reported as fractions (0.92, 0.95, 0.94) while the same metric is reported as a percentage in Tables I, III, IV, and V (e.g., 94.26 in Table IV); please unify the scale and label the units explicitly.","section":"Table II"},{"comment":"The paper should specify how the two enrollment utterances E1 and E2 are selected for DSEF-PNet: whether they are random distinct utterances from the same speaker, how the 'random' duration policy in Table V is implemented, and whether the two enrollments are always different files. The current description is insufficient for exact reproduction.","section":"Section V-A and Eq. (3)"},{"comment":"Equation (4) defines the HEIT loss on complex spectrograms, while Eq. (5) uses the SISDR loss, which is normally defined on waveforms, with the same symbol 'Y^pse' for both domains. Please clarify how the waveform is obtained from the complex spectrogram and confirm that the L1 consistency is applied in the STFT domain.","section":"Section IV-B, Eqs. (4)-(5)"},{"comment":"The ratio M:N of PSE and SE samples in each unified mini-batch is not specified. Because this ratio controls the balance between the two tasks in USEF-PNet, it should be reported.","section":"Section IV-A, Eq. (1)"},{"comment":"The abstract and introduction motivate robustness to variations such as emotional tone and semantic content, but the experiments only vary enrollment duration (Table V). Either add experiments that vary these factors or temper the claims and explicitly state the limitation.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The USEF-PNet contribution is reasonably solid and could be acceptable with the baseline-configuration issue clarified. The DSEF-PNet central claim currently rests on a confounded comparison: the two-enrollment pairing and the HEIT loss are varied together, so the disentanglement attribution is not established. This is fixable with a lambda=0 ablation and a modest direct diagnostic, which is why I recommend major revision rather than rejection. The authors should also check the architectural inconsistency between Sections III and V-B before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful incremental paper, not a breakthrough. The unified SE/PSE trick is neat — zeroing the enrollment vector during training lets one SEF-PNet handle both tasks without touching the architecture, and the Libri2Mix/VoiceBank numbers are in the right ballpark. The HEIT consistency loss for enrollment disentanglement is also simple and parameter-free, and the authors seem right that no prior work tackles disentanglement in embedding-free PSE. They report the negative LSEP result honestly instead of burying it.\n\nThe main contribution claim is the disentanglement mechanism, and there the paper has a real hole. DSEF-PNet differs from SEF-PNet in two ways at once: each mixture is paired with two enrollment utterances, and the lambda·LHEIT term is added. The gains in Table III could easily come from the doubled enrollment sampling acting as data augmentation and an enlarged effective batch. Without a lambda=0 ablation, attributing the improvement to the consistency constraint is not supported. The paper doesn't report lambda either, so the results are not reproducible as written. The disentanglement itself is also inferred from the training objective rather than measured — by construction the loss encourages identical outputs for two enrollments, so saying it isolates identity is close to restating the loss. That's an interpretive overclaim, but a fixable one: a lambda=0 row plus measuring something like enrollment content variation impact would settle it.\n\nSmaller issues: no error bars or significance tests on differences like SISDR 13.00 to 13.57; Table II's SE baseline appears not to state whether ISA was removed, which matters for fairness; and the reported gains are modest, with several metrics flat.\n\nHonestly, the empirical core probably holds up for the unified architecture — USEF-PNet matching or slightly beating task-specific models is a plausible and practically useful result. The disentanglement story needs more work. The paper deserves a serious referee but should come back with the missing ablation and clearer baselines.\n\nFor your reading group: maybe, if people care about PSE deployment. I'd cite it if I were writing about lightweight PSE, but I'd treat the HEIT claim with caution.","headline":"Useful incremental paper on unifying SE/PSE with a simple zero-enrollment trick, but the disentanglement mechanism needs a lambda=0 ablation before I'd trust the attribution.","tokens_in":19958,"tokens_out":1923,"would_cite":true,"duration_ms":20785,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One speaker-embedding-free network can perform both conventional and personalized speech enhancement, and an unsupervised consistency loss makes it insensitive to variation in the enrollment speech.","keywords":["personalized speech enhancement","speech enhancement","unified architecture","speaker embedding-free network","unsupervised speech disentanglement","enrollment robustness","long-short enrollment pairing","target speaker extraction"],"falsifier":"On the Libri2Mix 2-speaker condition, train DSEF-PNet with same-style enrollment pairs (both neutral read speech) and cross-style pairs (one neutral, one whispered or shouted) for the same speaker, matching content. If HEIT removes only identity-irrelevant variation, cross-style pairs should give the same SISDR as same-style pairs; a measurable drop would show the consistency term is discarding identity-relevant cues. The paper does not report this comparison or the value of $\\lambda$ in Eq. (5), so the experiment remains open.","tokens_in":18989,"feed_emoji":"🎙️","tokens_out":12167,"duration_ms":110647,"temperature":0.7,"pith_summary":"The paper aims to establish that a single speaker-embedding-free architecture, previously built for personalized speech enhancement (PSE), can also handle conventional speech enhancement (SE) with no extra parameters, and that a simple unsupervised consistency constraint can make PSE robust to changes in the enrollment utterance. For the unified model USEF-PNet, the authors train the same network with either a real enrollment utterance or a zero-valued dummy enrollment, so that SE becomes the special case of PSE with no reference. For disentanglement, DSEF-PNet pairs each noisy mixture with two different utterances from the same target speaker and forces the two enhanced outputs to match, arguing that this removes emotion, wording, and other identity-irrelevant factors while preserving speaker identity. If these claims hold, one lightweight model could replace two task-specific systems and tolerate the messy enrollment speech found in real applications.","feed_headline":"One network handles both conventional and personalized speech enhancement","feed_subtitle":"Zero enrollment means denoising; real enrollment means speaker extraction; a consistency loss removes emotion and wording.","key_machinery":"The load-bearing object is the heterogeneous enrollment invariant training (HEIT) loss, $\\lambda \\|\\hat{Y}^{\\mathrm{pse}}_1 - \\hat{Y}^{\\mathrm{pse}}_2\\|_1$, minimized alongside two SI-SDR losses in Eq. (5). It carries the disentanglement claim: because the two enrollments share only the target speaker's identity, forcing the outputs to agree is supposed to strip away emotion, speaking rate, wording, and noise while leaving identity untouched. The companion mechanism is the zero-dummy-enrollment batch construction in Eq. (1), which lets conventional SE be processed by the same network by concatenating a zero vector in place of enrollment features; this is what turns PSE into a unified task without changing the architecture. Both operate on the speaker-embedding-free SEF-PNet backbone, whose interactive speaker adaptation module feeds enrollment context directly into the time-frequency stream.","core_discovery":"The central claim is that the SEF-PNet backbone already carries enough information to serve both tasks; what changes is only how it is trained. USEF-PNet constructs each mini-batch from PSE samples (mixture plus real enrollment) and SE samples (mixture plus a zero dummy enrollment), trains with the negative scale-invariant signal-to-distortion ratio (SI-SDR) loss, and reports performance on Libri2Mix and VoiceBank-DEMAND that matches or exceeds the task-specific SEF-PNet baselines. DSEF-PNet adds heterogeneous enrollment invariant training (HEIT): the same mixture is passed through the network once with each of two same-speaker enrollments, and the L1 distance between the two complex-spectrogram outputs is added to the two SI-SDR losses, so the model is pushed to produce the same enhancement regardless of which enrollment it sees. The paper reports that this improves the Libri2Mix 2-speaker condition from 13.00 to 13.57 dB SISDR and 3.01 to 3.08 PESQ, and that combining both strategies (UDSEF-PNet) improves both the PSE and SE tasks further. The authors also explore long-short enrollment pairing (LSEP), and find that random enrollment durations during training generalize better than this pairing.","pith_inferences":["Beyond the paper, the zero-dummy-enrollment trick should transfer to any PSE model that concatenates an enrollment representation to the mixture, because the paper's rationale depends only on that concatenation, not on the SEF-PNet specifics.","Beyond the paper, HEIT's invariance is demonstrated only by final enhancement metrics, so the natural next test is a direct probe of the learned representation: if the internal enrollment features of whispered and shouted versions of the same speaker collapse to the same point, then true identity disentanglement is happening; if only the outputs collapse, the model may simply be averaging over enr","Beyond the paper, LSEP's negative result suggests a testable modification: instead of a hard L1 tie between short and long enrollment outputs, a soft or duration-conditioned alignment might preserve the extra speaker detail that short enrollments need."],"forward_implications":["A single USEF-PNet checkpoint can replace separate SE and PSE systems: on the full Libri2Mix 2-speaker condition it matches SEF-PNet's SISDR of 13.00 dB, while on VoiceBank-DEMAND it improves SE SISDR from 19.21 to 19.69 dB.","PSE robustness to changes in the enrollment utterance does not cost anything at inference: DSEF-PNet uses the same weights and runtime as SEF-PNet, and improves the 2-speaker condition SISDR from 13.00 to 13.57 dB.","The two ideas compose: UDSEF-PNet, which trains one SEF-PNet architecture with both tricks, reports 13.34 dB SISDR on the 2-speaker condition and 19.95 dB SISDR on VoiceBank-DEMAND, better than either trick alone.","Training with randomly chosen enrollment durations generalizes better than always pairing a short with a long utterance or always using long enrollments, so duration variability during training is itself a robustness lever."],"supporting_citations":[{"why":"Provides the SEF-PNet backbone with ISA and LCA modules, plus the PSE baseline numbers that USEF-PNet and DSEF-PNet extend without changing the architecture.","marker":"[26]"},{"why":"Supplies the sDPCCN backbone and the encoder/decoder/TCN configuration on which SEF-PNet is built.","marker":"[12]"},{"why":"Defines the negative scale-invariant signal-to-distortion ratio (SI-SDR) loss used in every training objective in the paper.","marker":"[49]"},{"why":"Provides the Libri2Mix dataset and its three PSE evaluation conditions, including the 2-speaker condition used for the main comparisons.","marker":"[50]"},{"why":"Provides the VoiceBank-DEMAND dataset used for the conventional SE task in the unified-architecture experiments.","marker":"[51]"},{"why":"Identifies the target-confusion problem caused by enrollment emotion and prosody, which motivates the HEIT disentanglement mechanism.","marker":"[23]"},{"why":"Frames the unified-enhancement goal that motivates building one model to handle multiple speech enhancement tasks.","marker":"[19]"},{"why":"Exemplifies prior enrollment disentanglement with multiple encoder branches and staged losses, the complexity the paper's single-consistency-term method is meant to avoid.","marker":"[24]"}],"fun_headline_variants":["One network for both denoising and speaker extraction","Consistency loss strips emotion from enrollment, boosting PSE","Zero-enrollment plus real enrollment trains one model for both tasks","Two enrollments, one output: disentangle speaker from emotion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise, which enters in Eqs. (3)–(5), is that two different enrollment utterances of the same speaker share only the speaker's identity, so the HEIT consistency loss forcing their outputs to be equal removes only emotion, wording, and other irrelevant factors and never discards useful speaker information; the paper tests this only through downstream enhancement metrics and does not report the value of $\\lambda$.","fun_headline_variants_meta":{"raw":{"variants":["One network for both denoising and speaker extraction","Consistency loss strips emotion from enrollment, boosting PSE","Zero-enrollment plus real enrollment trains one model for both tasks","Two enrollments, one output: disentangle speaker from emotion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00097,"raw_usage":{"total_tokens":4211,"prompt_tokens":1114,"completion_tokens":3097,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":730,"completion_tokens_details":{"reasoning_tokens":3029}},"tokens_in":730,"tokens_out":3097,"duration_ms":23441,"temperature":1.0,"reasoning_tokens":3029,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:36:35.991180+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the Libri2Mix 2-speaker condition, train DSEF-PNet with same-style enrollment pairs (both neutral read speech) and cross-style pairs (one neutral, one whispered or shouted) for the same speaker, matching content. If HEIT removes only identity-irrelevant variation, cross-style pairs should give the same SISDR as same-style pairs; a measurable drop would show the consistency term is discarding identity-relevant cues. The paper does not report this comparison or the value of $\\lambda$ in Eq. (5), so the experiment remains open.","supporting_citations":[{"cited_title":"SEF-PNet: Speaker encoder- free personalized speech enhancement with local and global contexts aggregation,","cited_arxiv_id":null,"evidence_quote":"Provides the SEF-PNet backbone with ISA and LCA modules, plus the PSE baseline numbers that USEF-PNet and DSEF-PNet extend without changing the architecture."},{"cited_title":"DPCCN: Densely-connected pyramid complex convolutional network for robust speech separation and extraction,","cited_arxiv_id":null,"evidence_quote":"Supplies the sDPCCN backbone and the encoder/decoder/TCN configuration on which SEF-PNet is built."},{"cited_title":"Sdr–half-baked or well done?","cited_arxiv_id":null,"evidence_quote":"Defines the negative scale-invariant signal-to-distortion ratio (SI-SDR) loss used in every training objective in the paper."},{"cited_title":"Investi- gating rnn-based speech enhancement methods for noise-robust text-to- speech,","cited_arxiv_id":null,"evidence_quote":"Provides the VoiceBank-DEMAND dataset used for the conventional SE task in the unified-architecture experiments."},{"cited_title":"Target confusion in end-to-end speaker extraction: Analysis and approaches,","cited_arxiv_id":null,"evidence_quote":"Identifies the target-confusion problem caused by enrollment emotion and prosody, which motivates the HEIT disentanglement mechanism."},{"cited_title":"Urgent Challenge: Universality, robustness, and generalizability for speech enhancement,","cited_arxiv_id":null,"evidence_quote":"Frames the unified-enhancement goal that motivates building one model to handle multiple speech enhancement tasks."},{"cited_title":"Self-supervised disentangled representation learning for robust target speech extraction,","cited_arxiv_id":null,"evidence_quote":"Exemplifies prior enrollment disentanglement with multiple encoder branches and staged losses, the complexity the paper's single-consistency-term method is meant to avoid."}],"review_version":1}