{"id":"56572efe-29df-4b20-aecb-10f77b0efcf6","arxiv_id":"2411.19770","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Noro adds a dual-branch reference encoder and a contrastive speaker loss so one-shot voice conversion stays accurate when the reference clip is noisy, and its reference encoder doubles as a competitive self-supervised speaker encoder.","lead":"Noro hardens one-shot voice conversion against noisy reference audio by training a dual-branch speaker encoder with a contrastive loss, keeping converted speech intelligible where a standard baseline degrades. The paper also shows that a voice conversion reference encoder can double as a self-supervised speaker encoder, reaching a 5.32% equal error rate on SUPERB speaker verification.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported noise robustness may come from ordinary noisy-reference data augmentation rather than the proposed dual-branch module and contrastive loss, because Noro is trained on augmented references while the baseline is not; no control ablation separates these causes.","rationale":"The reader's verdict is CONDITIONAL, and my concern does not change that: the paper should be accepted only after the missing control ablation and related analyses are supplied. The reader's weakest_assumption focuses on the train-inference mismatch and transfer to unseen noise types; my concern is closely related but more fundamental. The train-inference mismatch is real, but it is secondary to the absence of a training-strategy control. If a simpler single-branch augmented baseline matches Noro, then the dual-branch module and contrastive loss are unnecessary for the headline result, and the central claim of 'innovative components' is not established. If the control does not match Noro, the mismatch concern becomes more relevant but is still testable. I therefore do not move the verdict, but I sharpen the required experimental condition: a single-branch data-augmentation control is the decisive experiment.","tokens_in":7946,"tokens_out":2780,"duration_ms":29122,"concrete_test":"Run a minimal control experiment: take the pre-trained baseline, apply the same DEMAND noise augmentation with SNR drawn from N(0,20) dB to the reference segment only, and fine-tune with only the diffusion loss Ldiff, keeping the same segmentation, batch size, and training steps as Noro. Do not use the dual-branch module or Lref. Evaluate this single-branch augmented baseline on the same unseen-noise test set at 0-5 dB and 5-10 dB SNR. If its CER and SECS are close to Noro's (within about 0.5 CER), then the proposed dual-branch module and contrastive loss are not the cause of the robustness gain. If its CER is at least 2 points worse than Noro's 4.66, the components are supported. Also report the same control under clean conditions to check for the claimed 'minor reduction'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the dual-branch reference encoder and the noise-agnostic contrastive speaker loss are what make Noro robust in noisy conditions (Sec. II-B). But the experimental protocol cannot support this attribution. Noro is initialized from the pre-trained baseline and then trained using noisy references generated by mixing DEMAND noise at SNRs drawn from a normal distribution (Sec. II-B.1). The baseline was trained only on clean references. Therefore, any training that exposes the reference encoder to noisy inputs might improve noise robustness, independent of the dual-branch averaging and Lref. The paper reports only Noro versus the clean-trained baseline and other non-augmented methods; it does not report a single-branch baseline retrained with the same augmentation and the same diffusion loss, nor an ablation with Lref removed. There is also a train-inference mismatch: during training the diffusion model is conditioned on the average of the clean and noisy branch outputs, but at inference it is conditioned on a single branch output (Sec. II-B.1). The t-SNE visualization suggests the branches produce similar representations, but it does not establish that the score network behaves identically on averaged versus single-branch conditioning inputs. Without the control ablation, the headline improvement in CER/SECS at 0-5 dB SNR cannot be uniquely credited to the proposed components.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Noro, a noise-robust one-shot voice conversion system built on a diffusion-based baseline. Noro replaces the baseline's reference encoder with a dual-branch module that shares weights, processes clean and noisy reference speeches, and feeds their averaged representation to the acoustic model during training; it also adds a noise-agnostic contrastive speaker loss. In a second contribution, the authors repurpose the baseline reference encoder as an SSL speaker encoder (VC-SPK2VEC) and evaluate it on the SUPERB speaker verification task. Experiments compare Noro against the baseline and three one-shot VC systems under clean and noisy conditions, reporting objective (CER, SECS) and subjective (CMOS, SMOS) metrics, plus a t-SNE visualization of reference representations.","tokens_in":8216,"tokens_out":3752,"duration_ms":33300,"significance":"If the central claims hold, Noro offers a simple training strategy that substantially improves one-shot VC robustness to noisy reference speech, and the hidden speaker representation result is a noteworthy finding that connects VC pretraining to SSL speaker encoders. The paper is well-motivated and builds on public code and datasets (Amphion, LibriLight, VCTK, DEMAND), which is a strength. However, the experimental design currently does not isolate the contribution of the proposed dual-branch module and contrastive loss from the effect of noisy-reference data augmentation alone, and it omits comparisons with the noise-robust VC methods cited in the introduction. These gaps weaken the attribution of the reported gains and the claimed state-of-the-art relevance.","major_comments":[{"comment":"The experimental protocol cannot uniquely credit the proposed components with the observed robustness improvement. Noro is initialized from the pre-trained baseline and then trained with noisy reference speeches generated by mixing DEMAND noise at SNRs drawn from a normal distribution, while the baseline was trained only on clean references. Therefore the improvement at 0-5 dB (CER 4.66 vs 7.26, SECS 80.09 vs 77.28) could result from ordinary noisy-reference data augmentation alone, independent of the dual-branch averaging and Lref. The paper reports no control ablation with a single-branch encoder trained on the same augmented references and the same diffusion loss, nor an ablation that removes the contrastive loss. Without such controls, the central attribution in the abstract and Section II-B is not established.","section":"Section II-B.1 and Table I"},{"comment":"There is a train-inference mismatch: during training the diffusion model is conditioned on the average of href and h'ref, but at inference it is conditioned on a single branch output. The t-SNE visualization shows that the clean and noisy branch outputs are similar for Noro, but it does not establish that the score network behaves identically when fed the average versus a single branch output. The authors should either provide a theoretical or empirical argument that the mismatch is benign, or modify the training procedure (e.g., randomly dropping one branch during training) to align conditioning between training and inference.","section":"Section II-B.1 and Fig. 2"},{"comment":"The introduction cites several existing noise-robust VC methods, including data augmentation approaches [7], domain adversarial training [12], and conditional denoising [13], but the evaluation does not compare against any of them. Since the paper claims that Noro improves robustness in challenging real-world conditions, a comparison with at least the most related prior work under the same protocol is necessary to assess the contribution. The current comparison only includes FaCodec-VC, FreeVC, DiffVC, and the author's own baseline, none of which were specifically designed for noisy reference robustness.","section":"Section III-A.1 and Tables I-II"},{"comment":"The evaluation lacks statistical rigor: the subjective tests use only 10 pairs and 12 listeners, and the objective metrics are reported as point estimates without error bars or significance tests. In clean conditions, Noro and the baseline are nearly identical (CER 4.74 vs 4.71), and the claimed subjective advantage in noisy conditions (CMOS 2.95 vs 2.09) could be sensitive to listener variability. The authors should report confidence intervals, per-utterance standard deviations, or significance tests, and ideally increase the number of subjective evaluation items.","section":"Section III-A and Tables I-II"}],"minor_comments":[{"comment":"Reference [23] is identical to reference [10]; one of the two duplicates should be removed or replaced with the intended source.","section":"References"},{"comment":"The test set is described as using 'unseen types' of noise, but the specific noise types and their proportions are not listed; reporting per-condition results would help the reader judge the generality of the robustness transfer.","section":"Section III-A.2 and Table I"},{"comment":"The diffusion loss equation lacks explicit norms and a clear dependence of the score network on the segment length; rewriting it with proper L1 norm notation would improve clarity.","section":"Section II-A.3"},{"comment":"The comparison in Table III mixes models trained on different amounts of data and with different architectures; a sentence noting that this is not a strictly controlled comparison would be useful for interpreting the EER differences.","section":"Section III-B.2 and Table III"},{"comment":"The definition of the contrastive loss uses CrossEntropy with a mask Mi,j; it should be clarified whether the diagonal and the positive pairs are excluded from the denominator, as is standard in supervised contrastive learning.","section":"Section II-B.2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is plausible and the empirical trend is promising, but the missing ablation and the train-inference mismatch are load-bearing issues that must be addressed before the paper can be considered for publication. The speaker representation result is a nice side finding, though it is not fully integrated with the main story. The duplicate reference [10]/[23] and the absence of comparisons with cited noise-robust methods should also be fixed. With the control experiments and a more rigorous evaluation, the paper could become a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The bottom line: Noro is a clean, well-written study of a genuinely important problem—one-shot VC with noisy reference speech—and the headline result is clearly demonstrated against the authors' own baseline. At 0–5 dB SNR, Noro holds CER at 4.66 and SECS at 80.09 while the baseline degrades to 7.26 and 77.28. The gain is real. The problem is that the experiment does not isolate what causes it.\n\nWhat is new: the specific combination of a shared-weight dual-branch reference encoder, a noise-agnostic contrastive speaker loss, and a diffusion VC backbone. Also genuinely interesting is the VC-SPK2VEC byproduct: the baseline reference encoder, repurposed as a speaker encoder, gets 5.32% EER on VoxCeleb1 under SUPERB, competitive with HuBERT Base and better than several larger SSL models. That result is evaluated on a held-out benchmark and gives the paper an independent hook. The code is public through Amphion, which is a plus.\n\nThe soft spot is exactly what the stress-test note identifies: no control ablation. Noro is initialized from the pre-trained baseline and then trained with noisy references from DEMAND at random SNRs. The baseline was trained on clean references only. So any fine-tuning that exposes the reference encoder to noise—even a single-branch encoder with the same diffusion loss—might reproduce most of the improvement. The paper never reports a single-branch augmented baseline, nor an ablation with Lref removed. That means the specific claims about the dual-branch module and the contrastive loss are not yet supported. The t-SNE showing better clean/noisy mixing in Noro is suggestive, but it again compares against the clean-only baseline.\n\nOther weaknesses, in proportion: no comparison with the noise-robust VC methods cited in the introduction ([7], [9], [12], [13]); no error bars or significance tests; the subjective evaluation uses only 10 pairs and 12 listeners. The train-inference mismatch—averaged conditioning during training, single-branch at inference—is noted but never analyzed. Any one of these would be minor; together they make the paper's central attribution feel fragile.\n\nThis deserves peer review, not desk rejection. The problem is important, the writing is clear, and the missing control is easy to add. A referee should ask for: (1) a single-branch encoder trained with the same augmentation, with and without Lref; (2) comparisons with prior noise-robust methods; (3) error bars or significance tests; (4) a few words on why the train-inference mismatch is benign. With those additions, this could be a solid contribution. As it stands, accept only conditionally.","headline":"A competent, readable paper on a real problem whose central attribution is undercut by a missing ablation: the noise-robustness gain may come from simple reference-path augmentation, not the dual-branch encoder and contrastive loss.","tokens_in":8752,"tokens_out":2465,"would_cite":true,"duration_ms":22891,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Noise-agnostic reference embeddings keep one-shot voice conversion intelligible even when the target speaker's clip is noisy.","keywords":["one-shot voice conversion","noise robustness","dual-branch reference encoder","noise-agnostic contrastive speaker loss","speaker representation learning","self-supervised learning","diffusion model","speaker verification"],"falsifier":"A concrete falsifying experiment: evaluate Noro on reference clips corrupted by non-stationary noise types not in DEMAND (e.g., street babble or music) at SNR below 0 dB. If the character error rate jumps back to the baseline's noisy-condition level, the claimed noise-agnostic robustness is limited to the trained noise distribution; likewise, ablating the contrastive loss while keeping the dual-branch averaging would show how much of the gain comes from the loss itself.","tokens_in":7736,"feed_emoji":"🎙️","tokens_out":7753,"duration_ms":59043,"temperature":0.7,"pith_summary":"The paper argues that the standard approach of feeding a speaker-reference embedding from a noisy clip to a diffusion-based voice converter is the main failure point in real-world use, and that the failure can be fixed by making the reference embedding itself noise-invariant. Noro does this with a weight-shared dual-branch reference encoder that sees both a clean reference and a stochastically noise-corrupted copy, together with a contrastive speaker loss that forces the two branches to converge. At 0–5 dB SNR, Noro's character error rate stays near the clean level (4.66 versus 7.26 for the baseline) and speaker similarity improves substantially, while the clean-condition numbers are essentially unchanged. The same reference encoder, repurposed directly as a speaker encoder without fine-tuning, reaches a 5.32% equal error rate on a standard speaker-verification task, which the paper presents as evidence that one-shot VC training doubles as self-supervised speaker representation learning.","feed_headline":"Noisy reference clips no longer ruin one-shot voice conversion","feed_subtitle":"Dual-branch encoding plus a contrastive loss holds error rate flat at 0–5 dB SNR while keeping clean performance.","key_machinery":"The load-bearing mechanism is the dual-branch reference encoding module with the noise-agnostic contrastive speaker loss. Two transformer encoders share weights: one encodes the clean reference mel-spectrogram, the other encodes the same utterance corrupted by one of eight DEMAND noise types at an SNR sampled from a normal distribution with mean 0 dB and standard deviation 20 dB. The two utterance-level embeddings are averaged and then fed to the WaveNet diffusion model as conditioning, and a contrastive loss over the concatenated batch pulls clean and noisy embeddings of the same speaker together while pushing different speakers apart. At inference only one encoder branch is used, so the deployed model has the same structure and cost as the baseline. The paper attributes Noro's robustness to this noise-agnostic reference representation.","core_discovery":"The central discovery is that noise robustness in one-shot voice conversion can be trained into the reference representation rather than bolted on as enhancement. The paper's Noro system trains a dual-branch reference encoder with shared weights on clean and noisy versions of the same reference speech, feeding the average of the two representations to the diffusion model and supervising the branches with a noise-agnostic contrastive speaker loss. After this training, clean and noisy reference speech produce reference embeddings that are mixed in t-SNE space instead of separated by noise condition, and at inference the model uses only a single encoder branch. On the VCTK test set with unseen noise types at 0–5 dB SNR, Noro achieves CER 4.66 and SECS 80.09 versus the baseline's 7.26 and 77.28, while in clean conditions the two systems are nearly identical (CER 4.74 vs 4.71, SECS 82.38 vs 82.35). The paper also reports that the baseline reference encoder, evaluated without adaptation as a speaker encoder called VC-SPK2VEC, attains an EER of 5.32% on the SUPERB speaker-verification benchmark, beating several established self-supervised speech models.","pith_inferences":["The paper leaves the train-inference mismatch implicit: during training the diffusion model conditions on the average of clean and noisy reference embeddings, while at inference it sees only a single branch's output. Testing stochastic branch selection or clean-only conditioning during training would reveal whether this mismatch costs robustness.","The noise-agnostic contrastive loss is a general recipe for condition-invariant reference embeddings, so it could transfer to other reference-conditioned generative models (e.g., text-to-speech) or to robust speaker verification in noisy conditions.","The reported connection between VC quality and speaker-embedding quality suggests a mutual-training loop: improving the reference encoder for generation could improve downstream speaker tasks, and vice versa, but the paper does not test this synergy explicitly.","An obvious stress test beyond the paper's scope is SNR below 0 dB or non-stationary noise like cafeteria babble; the current normal(0,20) dB augmentation schedule covers moderate conditions and the paper's robustness claim would be stronger if it held there."],"forward_implications":["In noisy conditions (0–5 dB SNR), Noro's character error rate of 4.66 is nearly identical to its clean-condition 4.74, meaning intelligibility no longer collapses when the reference clip is degraded.","Speaker similarity under noise (SECS 80.09) approaches the clean-condition 82.38, so the converted voice keeps the target identity even from a noisy prompt.","Subjective scores (CMOS 2.95 vs 2.09 at 0–5 dB) confirm human listeners perceive Noro as substantially more natural than the baseline in noise.","Since the noisy test set uses noise types unseen in training, the improvement indicates the learned noise invariance generalizes beyond the augmentation set.","A one-shot VC reference encoder trained without speaker labels can serve as a competitive self-supervised speaker encoder (EER 5.32%), suggesting that large unlabeled speech corpora used for VC can also yield speaker representations."],"supporting_citations":[{"why":"Supplies the cross-attention reference encoder and WaveNet diffusion conditioning that the baseline and Noro build on.","marker":"[22]"},{"why":"Provides the frozen HuBERT semantic extractor for source content and serves as a comparison SSL model in the speaker-verification experiments.","marker":"[19]"},{"why":"Supplies the eight DEMAND noise types used to synthesize noisy reference speech for the dual-branch training.","marker":"[23]"},{"why":"Defines the SUPERB evaluation protocol under which VC-SPK2VEC's speaker-verification EER is measured.","marker":"[16]"},{"why":"Provides the WavLM SSL model whose speaker-verification results are compared with VC-SPK2VEC.","marker":"[20]"},{"why":"The DiffVC one-shot VC system is one of the state-of-the-art baselines Noro is compared against.","marker":"[1]"}],"fun_headline_variants":["Noise-proof one-shot voice conversion via hidden speaker learning","Dual-branch encoding tames noisy reference speech in VC","One-shot VC stays clean at 0 dB SNR with contrastive loss","Train noise robustness into the reference encoder, not the audio","Hidden speaker representations make one-shot VC noise-agnostic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that noise invariance learned from eight DEMAND noise types at SNRs centered on 0 dB transfers to unseen real-world noises, and that the train-time averaging of clean and noisy reference embeddings is harmless when only a single branch is used at inference; the paper does not analyze the latter mismatch explicitly.","fun_headline_variants_meta":{"raw":{"variants":["Noise-proof one-shot voice conversion via hidden speaker learning","Dual-branch encoding tames noisy reference speech in VC","One-shot VC stays clean at 0 dB SNR with contrastive loss","Train noise robustness into the reference encoder, not the audio","Hidden speaker representations make one-shot VC noise-agnostic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1341,"prompt_tokens":948,"completion_tokens":393,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":308}},"tokens_in":564,"tokens_out":393,"duration_ms":3951,"temperature":1.0,"reasoning_tokens":308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:48:43.738245+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifying experiment: evaluate Noro on reference clips corrupted by non-stationary noise types not in DEMAND (e.g., street babble or music) at SNR below 0 dB. If the character error rate jumps back to the baseline's noisy-condition level, the claimed noise-agnostic robustness is limited to the trained noise distribution; likewise, ablating the contrastive loss while keeping the dual-branch averaging would show how much of the gain comes from the loss itself.","supporting_citations":[{"cited_title":"Naturalspeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers,","cited_arxiv_id":null,"evidence_quote":"Supplies the cross-attention reference encoder and WaveNet diffusion conditioning that the baseline and Noro build on."},{"cited_title":"Hubert: Self- Supervised Speech Representation Learning by Masked Prediction of Hidden Units,","cited_arxiv_id":null,"evidence_quote":"Provides the frozen HuBERT semantic extractor for source content and serves as a comparison SSL model in the speaker-verification experiments."},{"cited_title":"Investigating RNN-Based Speech Enhancement Methods for Noise-Robust Text-to-Speech,","cited_arxiv_id":null,"evidence_quote":"Supplies the eight DEMAND noise types used to synthesize noisy reference speech for the dual-branch training."},{"cited_title":"SUPERB: Speech Processing Universal Performance Benchmark,","cited_arxiv_id":null,"evidence_quote":"Defines the SUPERB evaluation protocol under which VC-SPK2VEC's speaker-verification EER is measured."},{"cited_title":"Wavlm: Large- scale Self-supervised Pre-training for Full Stack Speech Processing,","cited_arxiv_id":null,"evidence_quote":"Provides the WavLM SSL model whose speaker-verification results are compared with VC-SPK2VEC."},{"cited_title":"Diffusion-Based V oice Con- version with Fast Maximum Likelihood Sampling Scheme,","cited_arxiv_id":null,"evidence_quote":"The DiffVC one-shot VC system is one of the state-of-the-art baselines Noro is compared against."}],"review_version":1}