{"id":"73b68ad7-9b09-4882-ae41-b9b6f4b0e64c","arxiv_id":"2412.20155","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Stable-TTS uses clean pretraining samples as prosody prompts plus a prior-preservation loss to make speaker-adaptive TTS robust to short and noisy target recordings.","lead":"This paper introduces Stable-TTS, a text-to-speech system that keeps producing clean, natural-sounding voice clones even when the only target recordings are short or noisy. It does this by borrowing stable 'prior' prosody from clean pretraining audio and adding a loss that stops fine-tuning from forgetting clean speech.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed benefit of prior prosody prompting is not robustly supported: the single hand-picked prior sample is never varied, and Table I's own ablation contradicts the paper's claim that removing it degrades all metrics.","rationale":"The reader's weakest_assumption focuses on a fixed hand-picked prior sample generalizing across texts and speakers. That is a real risk, but the paper's own ablation makes the concern sharper: the w/o Prior Prompt condition often matches or beats full Stable-TTS on WER and MOS, directly contradicting the paper's blanket claim that removing the component degrades all metrics. This internal inconsistency, together with the absence of any variation of the prior sample, means the central prosody-prompting mechanism is not yet empirically isolated. I do not think this warrants rejection: the full system does show large WER improvements over the two baselines, the prior-preservation loss ablation is supportive, and the proposed regularization is plausible. The gap is in attributing the improvement specifically to prior-sample prosody prompting, which is the paper's headline contribution. That gap reinforces the reader's CONDITIONAL verdict rather than changing it. The concrete test of randomizing prior samples would settle whether the effect is robust or cherry-picked, and would also address the missing significance testing because it yields a distribution over results.","tokens_in":8341,"tokens_out":9423,"duration_ms":96731,"concrete_test":"Re-run the full Table I evaluation (LibriTTS test-clean, VCTK, VoxCeleb; same fine-tuning budgets and evaluation protocols) using 10 held-out prior samples per gender randomly selected from LibriTTS-R clean-100/360, and also include the existing w/o Prior Prompt condition. Report mean and standard deviation of WER, MOS, SMOS, and SECS across prior samples. If WER or MOS varies strongly across prior samples, or if the target-prompt condition is not consistently worse than the best prior-sample condition, the prior-prosody-prompting claim is selection-dependent and the central mechanism is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism in Sections II-B and III-A is prior prosody prompting from a single 'carefully choose one speech sample for each male and female' prior sample, with no selection criteria and no sensitivity analysis. Yet Section III-C claims that removing any component degrades MOS, SMOS, and WER across all scenarios, which Table I does not support. Stable-TTS(w/o Prior Prompt), which uses target speech as the PLM prompt, achieves lower WER than full Stable-TTS on LibriTTS (0.83 vs 1.13) and VoxCeleb (1.02 vs 1.32), and higher MOS on LibriTTS (3.65 vs 3.37). Only SMOS improves consistently with the prior prompt. Thus the headline WER reductions over Grad-StyleSpeech and UnitSpeech do not actually require the prior-prosody-prompting component; the measured benefit of that component is mixed and could be an artifact of the unstated choice of one particular prior utterance per gender. Without varying the prior sample, the paper cannot support the causal role of prior-sample prosody in the claimed robustness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Stable-TTS, a speaker-adaptive text-to-speech framework that combines a prosody language model (PLM) with a prosody encoder, a timbre encoder, and a prior-preservation loss. The key idea is to use clean 'prior samples' from the pre-training corpus as the prosody prompt during inference and to add a prior-preservation loss during fine-tuning to prevent overfitting to noisy or limited target-speaker samples. The authors evaluate Stable-TTS on LibriTTS, VCTK, and VoxCeleb, reporting MOS, SMOS, WER, and SECS, and find large WER reductions relative to Grad-StyleSpeech and UnitSpeech while roughly preserving speaker similarity. They also report a fine-tuning data-scale experiment showing stable WER below 1% even with one target sample.","tokens_in":8555,"tokens_out":3799,"duration_ms":36435,"significance":"If the claimed results hold, Stable-TTS addresses a practically important problem: robust speaker adaptation under limited and noisy target speech. The reported WER improvements over strong baselines are substantial (81–92% relative reductions), and the architecture is a reasonable combination of existing components (Mega-TTS-style PLM, Grad-StyleSpeech diffusion backbone, DreamBooth-style prior-preservation loss). The paper includes a demo page, which is a strength. However, the central causal attribution—that prior-prosody prompting is what drives the robustness—is not cleanly supported by the paper's own ablation results, and the sensitivity of the method to the hand-picked prior sample is not examined. These issues must be addressed before the central claim can be accepted.","major_comments":[{"comment":"The text states that 'removing either component results in a degradation of performance in terms of MOS, SMOS, and WER across all scenarios,' but Table I contradicts this for the prior-prompt ablation. Stable-TTS(w/o Prior Prompt) achieves lower (better) WER than full Stable-TTS on LibriTTS (0.83 vs 1.13) and on VoxCeleb (1.02 vs 1.32), and higher MOS on LibriTTS (3.65 vs 3.37). Also, Stable-TTS(w/o PLM) gives higher SMOS on VoxCeleb (2.93 vs 2.65). Thus the ablation does not demonstrate that the prior-prosody-prompting component is necessary for the headline WER gains; the measured effect of that component is mixed and dataset-dependent. The authors should correct this claim, re-analyze the ablations, and discuss which component actually drives the improvement over baselines.","section":"Section III-C, Table I"},{"comment":"The prior sample used for the PLM prompt is described as 'carefully choose one speech sample for each male and female,' with no selection criteria, no description of how representative these samples are, and no sensitivity analysis. Since the entire prosody-prompting mechanism depends on the assumption that these two fixed samples provide prosody that transfers across arbitrary input texts and target speakers, the paper must vary the prior sample (e.g., multiple candidates per gender, different lengths, different prosodic content) and report WER/MOS/SMOS across those variations. Without this, the claimed robustness of prior-prosody prompting is not established, and the results could be artifacts of the specific chosen utterances.","section":"Section III-A, implementation details"},{"comment":"No statistical significance tests are reported for any of the subjective or objective metrics. Many of the differences that support the ablations are within the reported 95% confidence intervals; for example, on LibriTTS the MOS gap between Stable-TTS and Stable-TTS(w/o Prior Prompt) is 3.37±0.20 vs 3.65±0.21, and on VoxCeleb the SMOS gap between Stable-TTS and Stable-TTS(w/o PLM) is 2.65±0.23 vs 2.93±0.22. The authors should provide significance tests (e.g., paired bootstrap or Wilcoxon signed-rank) for the key comparisons, at least for the main result versus baselines and for the ablations, so that the reader can judge whether the claimed improvements are meaningful.","section":"Table I and Table III"}],"minor_comments":[{"comment":"The notation in Eq. (4) is slightly underspecified: the target prosody sequence p_t and the conditioning p_<t are not explicitly defined as the prosody codes aligned with the input phonemes µ_≤t; please clarify the indexing and the training objective for the PLM.","section":"Section II-B, Eq. (4)"},{"comment":"The paper reports SECS using Resemblyzer, which is a relatively weak speaker-verification model; please justify this choice or supplement it with a more standard speaker-embedding metric (e.g., ECAPA-TDNN cosine similarity).","section":"Section III-A, evaluation metrics"},{"comment":"The t-SNE visualization in the left panel is only qualitative and is not directly linked to WER or SECS; consider either removing it or adding a quantitative analysis of the representation space.","section":"Fig. 4"},{"comment":"The phrase 'a low band with rich prosodic contents of size 15' is vague; specify which mel bins are used and how the low band was selected.","section":"Section III-A, implementation details"},{"comment":"The paper states that 20 evaluators rated the samples, but it does not say whether they were expert or crowd-sourced, nor how they were recruited; please provide this information for reproducibility.","section":"Section III-A, subjective evaluation"}],"recommendation":"major_revision","confidential_remarks":"The ablation contradiction in Section III-C is the main substantive issue: the table does not support the blanket claim that removing prior prosody prompting degrades all metrics. This is not a mere presentation flaw, because the paper's central narrative attributes the robustness to the prior-prompt mechanism. A revision that corrects the claim, adds a sensitivity analysis of the prior sample, and includes significance tests would make the paper publishable; without those, the main contribution is not convincingly established. The paper's scope fits the journal; I do not see a circularity or novelty-disclosure concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Take: This is a competent, incremental speaker-adaptive TTS paper whose headline result—large WER reductions under limited and noisy samples—looks real, but whose central causal claim about prior prosody prompting is not backed by its own ablation table.\n\nWhat's new: the paper combines two known ideas in a new place: using clean pre-training samples as the prompt for the prosody language model (prior prosody prompting) and a DreamBooth-style prior-preservation loss during fine-tuning. That is a legitimate extension of Mega-TTS and DreamBooth, and it is the first application I know of to TTS fine-tuning. The full system beats Grad-StyleSpeech and UnitSpeech by a large margin on WER on all three datasets—roughly an 80–90% reduction—while keeping speaker similarity about level or better. The limited-data scaling experiment with 1, 5, 20, and 100 target samples is a nice addition; WER stays under 1% while Grad-StyleSpeech degrades badly.\n\nWhere the paper stumbles: the ablation in Table I directly contradicts the claim in Section III-C that removing any component degrades MOS, SMOS, and WER across all scenarios. Removing the prior prompt actually gives lower WER than the full model on LibriTTS (0.83 vs. 1.13) and VoxCeleb (1.02 vs. 1.32), and higher MOS on LibriTTS (3.65 vs. 3.37). Only SMOS improves consistently with the prompt. So the prior prosody prompting is not what drives the headline robustness result; the prior-preservation loss likely does most of the work. And the prompt is a single hand-picked utterance per gender, with no criteria and no sensitivity analysis—if that utterance changes, we have no idea what happens. No significance tests and only two baselines are also weaknesses. No code is released, which makes it harder to check the claims.\n\nThese are serious but fixable issues. The paper still shows a working system, and the prior-preservation loss transfer is a useful contribution. A competent referee could push the authors to vary the prior sample, report significance, and rewrite the ablation section to match their own numbers. That would make the paper much stronger. The current version overstates the role of the prior prompt and needs revision, but it deserves referee time.","headline":"A useful incremental TTS paper with a strong WER result, but the ablation overstates the role of prior prosody prompting.","tokens_in":9104,"tokens_out":4489,"would_cite":true,"duration_ms":40225,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Clean prosody from pre-training data keeps cloned voices clear.","keywords":["speaker-adaptive TTS","voice cloning","prosody prompting","diffusion model","prior-preservation loss","low-resource TTS","noisy speech robustness"],"falsifier":"A concrete test would be to take one of the two prior samples and synthesize a set of emotionally charged or prosodically varied sentences across many target speakers, then measure listening naturalness and WER; if the fixed prior sample makes all outputs sound like the prior speaker's intonation—for example, flat newsreader prosody for questions or exclamations—the central claim of prosody consistency fails. More directly, sampling many different prior samples and showing that WER and MOS swing widely would falsify the claim that a carefully chosen prior is sufficient.","tokens_in":8120,"feed_emoji":"🎙️","tokens_out":4559,"duration_ms":42609,"temperature":0.7,"pith_summary":"The paper aims to make speaker-adaptive text-to-speech reliable when only a few seconds of target speech are available, especially when that speech is noisy. It proposes Stable-TTS, which uses clean prior samples from the pre-training data as prosody prompts, keeping rhythm and intonation stable, while fine-tuning a diffusion model with a prior-preservation loss so the model does not forget how to produce clean audio. The central claim is that this combination keeps word error rates below 1% even with a single target sample and cuts word errors by roughly 90% on noisy recordings compared with two recent baselines, while roughly preserving speaker similarity.","feed_headline":"Voice cloning stays intelligible with just one noisy sample","feed_subtitle":"Stable-TTS borrows clean pre-training prosody to cut word errors by roughly 90% on noisy audio while keeping the target speaker's voice.","key_machinery":"The two load-bearing components are the prosody language model, an autoregressive model that predicts discrete phoneme-level prosody codes given a prompt speech sample, and the prior-preservation loss, a mean-squared-error term between the noise estimates of the frozen pre-trained diffusion model and the fine-tuned one computed on clean prior samples. The prior samples—a small hand-picked subset of the clean pre-training corpus, one male and one female voice—supply the prompt that drives the prosody language model and the anchor that the fine-tuned model must not drift away from.","core_discovery":"On the paper's own terms, the discovery is that prosody—the part of speech that carries rhythm, stress, and intonation—can be decoupled from the target speaker's timbre and sourced from clean pre-training data, making speaker adaptation stable under conditions that break previous methods. Stable-TTS's prosody language model predicts discrete prosody codes from a clean prior sample rather than from the noisy target audio, and a prior-preservation loss anchors the fine-tuned diffusion model to the pre-trained distribution. The paper reports that this yields word error rates below 1% with as few as one target sample and around 90% relative WER reduction on the noisy VoxCeleb set compared with Grad-StyleSpeech and UnitSpeech, with speaker similarity largely intact.","pith_inferences":["Editorial extension: if the prior-prosody prompt genuinely generalizes, the same framework could be applied to cross-lingual speaker adaptation, where a prior sample's language provides prosody templates for new languages—something the paper does not test.","Editorial extension: the reliance on a single hand-picked prior per gender suggests a stress test: vary the prior sample across speakers, emotions, and speaking rates and measure WER and speaker-similarity variance; the paper's fixed choice leaves this sensitivity unquantified.","Editorial extension: the prior-preservation loss is a direct analogue of a regularization technique from image generation, so one could expect it to also help other fine-tuning scenarios in audio, such as singing-voice synthesis or accent adaptation, though those domains are not demonstrated here."],"forward_implications":["On clean test sets, Stable-TTS reports word error rates below 1% even when fine-tuned on a single target sample, whereas a baseline's WER rises to nearly 49% in the same setting.","On the noisy VoxCeleb set, the reported WER is roughly 92% lower than that of the stronger baseline and 93% lower than that of the weaker baseline, while speaker similarity scores stay within about two to three points.","Fine-tuning remains stable across step counts, with 500 steps described as the sweet spot, so practitioners do not need careful early stopping.","The model works in a zero-shot setting as well as after fine-tuning, meaning the same weights can be deployed before any target-speaker data arrives.","Removing either the prosody language model or the prior-preservation loss degrades MOS, SMOS, and WER in every setting tested, so both components are presented as necessary."],"supporting_citations":[{"why":"Supplies the base any-speaker adaptive diffusion TTS architecture that Stable-TTS extends.","marker":"[3]"},{"why":"Introduces the prosody language model and discrete prosody codes used for prosody prompting.","marker":"[4]"},{"why":"Provides the prior-preservation loss formulation, adapted from personalized text-to-image diffusion.","marker":"[11]"},{"why":"Underlies the diffusion model and noise-estimation training objective.","marker":"[12]"},{"why":"Provides the clean LibriTTS-R pre-training corpus from which prior samples are drawn.","marker":"[22]"},{"why":"Supplies the noisy VoxCeleb evaluation set used to demonstrate robustness.","marker":"[9]"}],"fun_headline_variants":["Prosody from clean samples stabilizes voice cloning","One noisy sample, <1% word errors: Stable-TTS","Clean prosody, noisy voice: TTS adapts robustly","Borrow clean prosody, keep speaker timbre","Stable-TTS: decouple prosody to clone voices reliably"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach rests on the assumption that a single fixed clean prosody sample per gender can supply prosody that transfers to any input text and any target speaker without sounding mismatched.","fun_headline_variants_meta":{"raw":{"variants":["Prosody from clean samples stabilizes voice cloning","One noisy sample, <1% word errors: Stable-TTS","Clean prosody, noisy voice: TTS adapts robustly","Borrow clean prosody, keep speaker timbre","Stable-TTS: decouple prosody to clone voices reliably"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1357,"prompt_tokens":853,"completion_tokens":504,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":420}},"tokens_in":469,"tokens_out":504,"duration_ms":5676,"temperature":1.0,"reasoning_tokens":420,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:29:25.717848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be to take one of the two prior samples and synthesize a set of emotionally charged or prosodically varied sentences across many target speakers, then measure listening naturalness and WER; if the fixed prior sample makes all outputs sound like the prior speaker's intonation—for example, flat newsreader prosody for questions or exclamations—the central claim of prosody consistency fails. More directly, sampling many different prior samples and showing that WER and MOS swing widely would falsify the claim that a carefully chosen prior is sufficient.","supporting_citations":[{"cited_title":"Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models,","cited_arxiv_id":null,"evidence_quote":"Supplies the base any-speaker adaptive diffusion TTS architecture that Stable-TTS extends."},{"cited_title":"DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject- Driven Generation,","cited_arxiv_id":null,"evidence_quote":"Provides the prior-preservation loss formulation, adapted from personalized text-to-image diffusion."},{"cited_title":"Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech,","cited_arxiv_id":null,"evidence_quote":"Underlies the diffusion model and noise-estimation training objective."},{"cited_title":"LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus,","cited_arxiv_id":null,"evidence_quote":"Provides the clean LibriTTS-R pre-training corpus from which prior samples are drawn."},{"cited_title":"V oxCeleb: A Large- Scale Speaker Identification Dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the noisy VoxCeleb evaluation set used to demonstrate robustness."}],"review_version":1}