{"id":"673bc166-e12e-41ba-82de-8dcbf978b055","arxiv_id":"2507.14988","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Reinforcement learning on duration prediction improves intelligibility and speaker similarity in a 4-step distilled text-to-speech model, and teacher-guided sampling recovers prosodic diversity.","lead":"DMOSpeech 2 trains a speech-length predictor with reinforcement learning, using speaker similarity and word error rate as rewards, and adds a teacher-guided sampling step that restores output diversity in distilled text-to-speech models. If the results hold, high-quality zero-shot TTS can run several times faster than prior systems while matching or beating them on intelligibility and similarity.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline WER/SIM gains in Table 1 are not an independent test: the 1.5K-step checkpoint and group size were selected on a subset of Seed-TTS-en (A.2.1), the same set used for evaluation, so the RL-vs-no-RL comparison may be optimistically biased.","rationale":"The paper's strongest claim is an empirical one: GRPO on the duration predictor closes most of the gap to best-of-8 oracle and beats the no-RL student. For that claim to hold, the evaluation must be an unbiased estimate of performance. The paper reports exactly the kind of selection-on-the-test-set that undermines such estimates: A.2.1 describes validation on \"a small subset of the Seed-TTS-en evaluation set\" and choosing 1.5K steps and group size 16 from Figure 4; Table 1 then reports Seed-TTS-en as the evaluation result. The no-RL baseline is not selected this way, so the comparison is asymmetric. The problem is amplified by Figure 4 itself, which shows the metric degrading after about 2K steps; the reported result is a local optimum selected on the evaluation distribution, and there is no confidence interval to indicate stability. I do not think this is fatal: the authors may well be right, the code is promised, and the Chinese results plus the teacher-guided sampling provide some independent support. But the central quantitative claim about English WER/SIM cannot be fully assessed from the paper as written. A held-out evaluation and confidence intervals would settle it. This aligns with the reader's conditional verdict, though my emphasis is on test-set contamination rather than proxy fidelity per se; the two are related because an early-stopped proxy-optimized checkpoint is exactly where selection bias bites hardest.","tokens_in":19642,"tokens_out":6778,"duration_ms":76540,"concrete_test":"Before any further model selection, hold out a portion of Seed-TTS-en (or use a completely separate English test set such as LibriTTS or VCTK). Re-run the same GRPO training and evaluate checkpoints at 0.5K, 1K, 1.5K, 2K, and 3K steps on this held-out set. Report WER and SIM with confidence intervals for the selected checkpoint versus the no-RL duration predictor. If the 1.5K checkpoint no longer outperforms the no-RL baseline, or the gap shrinks below statistical significance, the central claim of substantial RL-driven improvement is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix A.2.1 states that the 1.5K-step checkpoint, group size, and related hyperparameters were selected using \"a small subset of the Seed-TTS-en evaluation set,\" with Figure 4 monitoring SIM/WER across 1K–6K steps and group sizes 8/16/32. The headline English numbers in Table 1 (WER 1.752 vs 3.750 for no-RL; SIM 0.698 vs 0.672) are then reported on the full Seed-TTS-en set, which includes the selection subset. Because early stopping and hyperparameters were chosen on the evaluation distribution, the reported improvement is not an unbiased estimate of the method's effect, and the no-RL baseline did not receive the same selection process. This is especially consequential because Figure 4 shows WER/SIM degrade sharply after roughly 2K steps, so the result rests on a fragile early-stopped checkpoint. The central claim that GRPO on the duration predictor substantially improves intelligibility and speaker similarity therefore may not generalize to unseen English speakers/texts, or to the same degree in other languages, even if the reward in Eq. (4) is a reasonable proxy. The fix is not to abandon RL but to demonstrate the effect on a genuinely held-out test set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DMOSpeech 2, an extension of the DMOSpeech framework that applies group relative policy optimization (GRPO) to the duration predictor of a flow-matching zero-shot TTS system, using speaker similarity and CTC-based word error rate as rewards. It also proposes teacher-guided sampling, a hybrid inference procedure that starts denoising with the teacher model and switches to the distilled student for the final steps, aiming to restore prosodic diversity lost during distillation. Experiments on Seed-TTS-en and Seed-TTS-zh compare the proposed system with the F5-TTS teacher, the student without duration-predictor RL, and several public baselines, reporting improved WER/CER and SIM at low real-time factors, plus CMOS listening test results.","tokens_in":19877,"tokens_out":4406,"duration_ms":46286,"significance":"If the reported gains hold under unbiased evaluation, the paper makes a solid contribution: it demonstrates that a previously untrained component of a non-autoregressive TTS pipeline (the duration predictor) can be optimized with a lightweight RL procedure, and it provides a simple sampling strategy that partially restores diversity in distilled diffusion models. The paper includes useful internal controls—a no-RL ablation, ground-truth durations, a best-of-8 oracle, and a teacher baseline—which strengthen the causal interpretation of the duration-predictor optimization. The promise of public code and models is commendable. However, the evaluation protocol contains a selection-bias issue that directly affects the headline quantitative claims, and the objective metrics lack uncertainty estimates, so the current version overstates the reliability of the improvements.","major_comments":[{"comment":"Hyperparameters and the early-stopping point (1.5K steps, group size 16) were selected using 'a small subset of the Seed-TTS-en evaluation set' (Appendix A.2.1, Figure 4), and the final results in Table 1 are then reported on the full Seed-TTS-en set, which includes that selection subset. The RL-vs-no-RL comparison (WER 3.750 vs 1.752; SIM 0.672 vs 0.698) is therefore optimistically biased, because the no-RL baseline did not receive the same selection process and the checkpoint was chosen on the evaluation distribution. This is especially consequential because Figure 4 shows sharp degradation after roughly 2K steps, making the result hinge on the selected early-stopped checkpoint. Please re-evaluate on a genuinely held-out test set that is disjoint from any hyperparameter or checkpoint selection (for example, split Seed-TTS-en into a validation subset and a disjoint test subset, or add an independent cross-sentence test set such as LibriTTS or VCTK), and report the no-RL baseline on the same held-out set. Also report bootstrap confidence intervals or significance tests for the objective metrics in Table 1, particularly for the small WER gap between DMOSpeech 2 (1.752) and Teacher-Guided (1.738).","section":"Appendix A.2.1 and Table 1"},{"comment":"The reward in Eq. (4) is computed with a CTC-based ASR model trained on Emilia, whereas the evaluation in Table 1 uses Whisper-large-v3 and Paraformer-zh on waveforms. The paper claims improved intelligibility but does not show that optimizing the CTC proxy transfers to the evaluation ASR models; the paper's own Figure 4 indicates that the proxy overfits after about 2K steps, so the final result depends on early stopping. Please provide evidence that the selected checkpoint's improvements are not an artifact of the CTC reward, for example by reporting reward and evaluation-metric trajectories on a validation set that is disjoint from the evaluation set, and by verifying the gains with a held-out ASR model that was not used in the reward.","section":"Eq. (4), Section 3.2.2"},{"comment":"Table 3 reports that Ground Truth Audio has a WER of 2.143 on Seed-TTS-en, which is higher than DMOSpeech 2's WER of 1.752 and also higher than Ground Truth Duration's WER of 1.821. This is counterintuitive: the reference audio used for evaluation should have low ASR error, and a higher WER for ground truth than for synthesized speech suggests either a transcription or evaluation artifact. Please clarify why the ground-truth recordings have such a high WER in this evaluation protocol, and discuss how this affects the interpretation of the WER differences among systems.","section":"Table 3, Section A.1"}],"minor_comments":[{"comment":"Typo: 'objecetive' should be 'objective' in the Baselines paragraph.","section":"Section 4.1"},{"comment":"Typo: 'Kullback-Liebler' should be 'Kullback-Leibler'.","section":"Appendix B.3"},{"comment":"The notation log p(x|C(y_k)) is ambiguous because C(y_k) is the ASR model's output sequence; please define C(·) precisely, e.g., as a function returning the log-likelihood of transcript x under the CTC model.","section":"Eq. (4)"},{"comment":"The figure would be clearer with a shared legend describing the three group sizes; the current legends inside each panel are redundant, and the vertical dashed line at 1.5K should be explicitly labeled in the caption.","section":"Figure 4"},{"comment":"The paper states that the student model uses 'half the batch size of the teacher model training' but does not give the actual batch size; please state the number of audio frames per batch. Also, Appendix A.2.2 refers to a 'validation set' for calibrating λ_SIM, but it does not specify whether that set is disjoint from the evaluation set; please clarify.","section":"Section 4.1 / Appendix A.2.2"},{"comment":"The abstract says 'reducing sampling steps by half without quality degradation,' but the comparison in Table 1 is between 32 teacher steps and 16 teacher-guided steps, while the student-only model uses 4 steps. Please clarify that the 'half' refers to the teacher-guided variant relative to the teacher, not to the student-only model.","section":"Abstract and Section 3.3.2"},{"comment":"The text first says 'a total of 320 samples in both English and Chinese' and later says '320 English and 320 Chinese samples'; please disambiguate the total number of evaluations.","section":"Appendix C"},{"comment":"The use of the queue Q and the update timing of π_old is somewhat unclear; please specify how many groups are accumulated before a policy update and when the old policy snapshot is refreshed.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The core technical idea is interesting and the internal controls are a strength, but the selection-bias issue in Appendix A.2.1 is load-bearing: the headline RL-vs-no-RL improvements are evaluated on a set that includes the subset used for early stopping and hyperparameter selection. I would like to see a genuinely held-out evaluation before endorsing the quantitative claims. The paper should also provide uncertainty estimates for the objective metrics and clarify the ground-truth WER anomaly in Table 3. If the authors can demonstrate the effect on a disjoint test set, the paper would be a solid acceptance candidate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Colleague],\n\nQuick take: this is a genuine extension—applying GRPO to the duration predictor rather than the whole generator—and it comes with better-than-usual internal controls: a no-RL ablation, ground-truth duration, a best-of-8 oracle, and a teacher baseline. The reward uses a CTC ASR while evaluation uses Whisper/Paraformer, which limits reward hacking concerns. The math is straightforward and the writing is clear.\n\nBut the headline WER/SIM improvement is not trustworthy as reported. In Appendix A.2.1 the authors admit they selected the 1.5K-step checkpoint, group size, and other hyperparameters on 'a small subset of the Seed-TTS-en evaluation set.' Then they report results on the full Seed-TTS-en set, which includes that subset. The no-RL baseline did not get the same selection process. Given that Figure 4 shows performance degrades sharply after ~2K steps, the early-stopped checkpoint is a fragile artifact of that selection. This is not a minor issue: the paper's central claim that RL on the duration predictor substantially improves intelligibility and similarity rests on it. The fix is straightforward—evaluate on a genuinely held-out test set or do proper nested validation.\n\nThere are also softer issues: Tables 1 and 2 lack confidence intervals, and some close comparisons (1.752 vs 1.738 WER) are within noise. The teacher-guided sampling contribution is interesting, but its subjective gains only reach significance for Chinese; the English naturalness gain is not significant and the objective metrics are essentially tied.\n\nThat said, the paper is worth engaging with. The idea of optimizing a small predictor rather than the whole generator is practical and likely to be picked up by others. The internal controls show the authors are thinking about confounds, even if the evaluation design undermines the headline. I'd send it to a serious referee, but with a strong request for a held-out evaluation before acceptance. For your reading group, it's a good case study in how evaluation-set-based hyperparameter selection can bias results.","headline":"A genuine extension—RL on the duration predictor—with good internal controls, but the headline gains are compromised by hyperparameter selection on the evaluation set itself.","tokens_in":20513,"tokens_out":2086,"would_cite":true,"duration_ms":22001,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Duration-predictor RL cuts WER from 3.75 to 1.75 in zero-shot TTS","keywords":["text-to-speech","duration prediction","reinforcement learning","GRPO","speaker similarity","word error rate","flow matching","metric optimization"],"falsifier":"Run the identical GRPO duration-predictor training on a held-out language or recording domain not represented in the training data and report WER and SIM before and after RL; if the gains vanish, the reward proxy, not duration optimization, is doing the work. A second check is to extend training past 2K steps; if performance does not degrade, the early-stopping story shown in Figure 4 is not reproducible.","tokens_in":19372,"feed_emoji":"🗣️","tokens_out":6795,"duration_ms":62042,"temperature":0.7,"pith_summary":"This paper claims that in diffusion-based zero-shot text-to-speech, the component that decides how long the speech will be is a major quality bottleneck and can be optimized directly. It treats the duration predictor as a stochastic policy and trains it with group-relative policy optimization, rewarding generated speech for low CTC word error and high speaker-embedding similarity. The result is a four-step student model that reaches a WER of 1.752 and SIM of 0.698 on the English test set, beating both the flow-matching teacher (1.947, 0.662) and the prior system without RL (3.750, 0.672) while running about five times faster than the teacher. The paper also introduces teacher-guided sampling, which uses the teacher for early denoising steps and the student for later ones, recovering most of the teacher's prosodic diversity that distillation had removed. If these claims hold, targeted RL on a single non-differentiable component is enough to close most of the gap to a best-of-8 oracle (WER 1.723, SIM 0.724) without end-to-end gradients.","feed_headline":"Duration-predictor RL cuts WER from 3.75 to 1.75 in zero-shot TTS","feed_subtitle":"Speaker similarity also beats the teacher, at five times the inference speed.","key_machinery":"The engine of the method is a GRPO loop around a transformer duration policy $P_\\phi(L_t|x,p_t)$ that outputs a distribution over 300 remaining-length classes of 100 ms each. For each input, sixteen durations are sampled with Gumbel-softmax temperature 0.7, speech is generated from each by the four-step student, and the reward $r_k = \\log p(x|C(y_k)) + \\lambda_{\\mathrm{SIM}} \\cdot \\cos(e_p, e_{y_k})$ is normalized into group advantages; a clipped importance-ratio loss with KL regularization toward a frozen reference policy updates the predictor. This machinery matters because duration prediction has no differentiable path to the generator, so the RL loop is what connects duration choices to measurable WER and speaker similarity. The second piece of machinery is teacher-guided sampling: the teacher denoises from noise down to a switching time $t_{\\mathrm{switch}}$, then the student completes the remaining steps, which restores early-step prosodic diversity that distillation compressed.","core_discovery":"The paper's central claim is that the duration predictor, not just the speech generator, can be aligned to perceptual metrics via reinforcement learning. The authors model remaining-length prediction as a policy and apply GRPO with groups of sixteen sampled durations, each rendered by the four-step distilled student and scored by the sum of a CTC ASR log-likelihood and a speaker-embedding cosine similarity term. This procedure lowers English WER from 3.750 to 1.752 and raises SIM from 0.672 to 0.698 compared with the same student before RL, and it beats the flow-matching teacher baseline (WER 1.947, SIM 0.662) while approaching the best-of-8 oracle (WER 1.723, SIM 0.724). The same pattern appears in Chinese CER and SIM. The paper further claims that teacher-guided sampling restores pitch variation from 30.3% below the teacher to 89.1% of the teacher's diversity while keeping a 1.8x speed advantage over full teacher inference.","pith_inferences":["Editorial inference: the reward's CTC term is trained on the Emilia corpus, so the reported gains likely reflect the proxy's own distribution; on unseen accents or recording conditions the WER advantage may shrink or reverse.","Editorial inference: Figure 4's sharp degradation after roughly 2K steps suggests the 1.5K checkpoint is an early-stopped optimum, so a production system would need a validation-based stopping rule rather than a fixed step count.","Editorial inference: because rewards are computed per full utterance, the method inherits any bias in the ASR and speaker-verification models; an alternative reward that penalizes duration outliers or uses reference-free intelligibility could trade some WER gain for robustness.","Editorial inference: the 100 ms duration bins impose a quantization floor on prosody; finer bins or continuous regression might unlock the remaining gap to the best-of-8 oracle."],"forward_implications":["If correct, non-autoregressive TTS systems can improve intelligibility and speaker similarity without backpropagating through the generator, because the RL signal comes from cheap four-step samples.","A 0.3B-parameter model with RL-optimized duration prediction can beat larger zero-shot baselines in WER and CER while running about five times faster than its own teacher, so scaling model size is not the only route to quality.","The near-oracle result (WER 1.752 versus best-of-8 1.723) implies duration prediction is close to saturated for this pipeline, leaving generator-side diversity and reward robustness as the next bottlenecks.","The teacher-guided variant shows that diversity and metric quality are not in conflict: mixing teacher early steps with student later steps recovers pitch variation while keeping the WER and SIM gains.","The same targeted-RL recipe could be applied to other generative components that are hard to differentiate, such as the teacher model itself, as the paper's conclusion suggests."],"supporting_citations":[{"why":"Establishes direct metric optimization for the speech generator and serves as the \"without RL\" baseline that DMOSpeech 2 extends.","marker":"[28]"},{"why":"Supplies the flow-matching teacher model and the speaking-rate-based duration heuristic used as a baseline.","marker":"[27]"},{"why":"Provides evidence that input durations significantly affect speaker similarity and word error rate, motivating duration optimization.","marker":"[24]"},{"why":"Supplies the encoder-decoder transformer architecture used for the duration predictor.","marker":"[26]"},{"why":"Provides the GRPO algorithm with clipping and KL regularization that the duration-policy training is built on.","marker":"[41]"},{"why":"Prior application of GRPO to flow-matching TTS, cited as the relevant RL approach for this family of models.","marker":"[38]"},{"why":"Provides the improved distribution matching distillation procedure used to train the student and the source of its reduced output diversity.","marker":"[40]"},{"why":"Supplies the Emilia training data used for the teacher, student, duration predictor, and CTC reward model.","marker":"[43]"}],"fun_headline_variants":["Duration-predictor RL cuts WER by half in zero-shot TTS","RL for duration prediction improves WER, speaker similarity in TTS","Teacher-guided sampling restores pitch diversity at 1.8x speed","RL on duration predictor improves WER from 3.75 to 1.75"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the CTC ASR likelihood used as the intelligibility reward is a faithful, non-gameable proxy for real intelligibility, and that the reported gains survive the early stopping at 1.5K steps before the reward overfitting the paper itself shows around 2K steps.","fun_headline_variants_meta":{"raw":{"variants":["Duration-predictor RL cuts WER by half in zero-shot TTS","RL for duration prediction improves WER, speaker similarity in TTS","Teacher-guided sampling restores pitch diversity at 1.8x speed","RL on duration predictor improves WER from 3.75 to 1.75"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000546,"raw_usage":{"total_tokens":2621,"prompt_tokens":968,"completion_tokens":1653,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":1571}},"tokens_in":584,"tokens_out":1653,"duration_ms":12006,"temperature":1.0,"reasoning_tokens":1571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:43:12.847855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical GRPO duration-predictor training on a held-out language or recording domain not represented in the training data and report WER and SIM before and after RL; if the gains vanish, the reward proxy, not duration optimization, is doing the work. A second check is to extend training past 2K steps; if performance does not degrade, the early-stopping story shown in Figure 4 is not reproducible.","supporting_citations":[{"cited_title":"DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis","cited_arxiv_id":"2410.11097","evidence_quote":"Establishes direct metric optimization for the speech generator and serves as the \"without RL\" baseline that DMOSpeech 2 extends."},{"cited_title":"Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation","cited_arxiv_id":null,"evidence_quote":"Supplies the Emilia training data used for the teacher, student, duration predictor, and CTC reward model."}],"review_version":1}