{"id":"86028138-8170-4433-a35b-618bc1d236b2","arxiv_id":"2412.11795","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ProsodyFM adds a phrase-break encoder, a break-duration predictor, and a bank of intonation-shape tokens to a flow-matching TTS backbone, improving prosody and intelligibility without explicit prosodic labels.","lead":"ProsodyFM is a text-to-speech system that learns to control phrase breaks and terminal intonation without manual prosody labels, reporting better intelligibility and prosody scores than four existing systems. It matters because unnatural pauses and flat intonation are a main source of robotic-sounding synthesized speech, and the model also offers fine-grained manual control over both.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The F1break metric is circular: PSST generates the training labels for break prediction and also scores the synthesized breaks, so ProsodyFM's phrasing advantage over baselines may reflect overfitting to PSST rather than better phrasing.","rationale":"The reader's weakest_assumption identifies the PSST circularity, and I agree this is the single most load-bearing concern. The phrase break metric is the only objective measure of the phrasing component of the central claim, and it is constructed from the same model that supervises the break predictor and break encoder. This is not a minor artifact; it directly affects the headline result. The paper does provide independent support: WER and MOSintelligibility are not affected by the circularity and still favor ProsodyFM, and the MOSbreak human judgment gives some independent evidence for phrasing, albeit on a small set. The ablation study in Table 4 also shows internal consistency. However, the objective F1break advantage and the causal interpretation 'better phrasing leads to better intelligibility' are weakened. I would not reject the paper because the subjective and intelligibility results are still positive, but the claim as stated needs an independent break annotation to be fully credible. I also note the sign error in Equation 9 (the 'Text-pitch Alignment Loss' is defined as the negative of an L2 distance, so minimizing it would maximize dissimilarity), which is a concrete internal inconsistency worth fixing and verifying against the released code. The out-of-distribution experiment in Table 3 compares only against MatchaTTS, further limiting the generalizability claim. These secondary issues, together with the circular metric, support a CONDITIONAL verdict requiring revision. Since the reader already recommended CONDITIONAL, my assessment does not change the verdict.","tokens_in":15123,"tokens_out":3856,"duration_ms":41033,"concrete_test":"Take a random subset of the LibriTTS test utterances (e.g., 50 per system) synthesized by ProsodyFM, MatchaTTS, StyleTTS2, StyleSpeech, and GenerSpeech. Have two trained phoneticians independently annotate phrase breaks (using ToBI break indices or perceptually marked pauses) on both the ground-truth and synthesized audio, blind to system identity. Compute F1break using the human annotations as reference instead of PSST, and compare the systems. If ProsodyFM no longer ranks first, or the margin over baselines shrinks to within inter-annotator agreement, the objective phrasing claim in Table 1 is not supported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The objective phrasing result (F1break in Table 1) is computed with PSST as the break detector on both ground-truth and synthesized speech (see 'Objective Evaluation Metrics'). Meanwhile, the Phrase Break Encoder uses a frozen PSST to identify breaks during training, and the T5 Phrase Break Predictor is fine-tuned with PSST-derived labels as ground truth (see 'Experimental Details'). Thus PSST is simultaneously the teacher that shapes ProsodyFM's break placements and the evaluator that measures them. The four baseline systems were not trained with PSST, so their breaks are not optimized to match PSST's boundary criteria. Any systematic bias in PSST—e.g., a preference for particular pause durations, a tendency to miss short breaks, or sensitivity to speaking rate—would inflate ProsodyFM's F1break relative to the baselines by construction. The subjective MOSbreak in Table 2 provides human evidence for better break similarity, but it uses only 15 utterances and labels derived from reference speech, so it is weaker than a dedicated independent annotation study. The central claim that ProsodyFM 'effectively improves the phrasing aspects of prosody' therefore rests on a metric whose validity is compromised by train/evaluation leakage. This is a load-bearing concern because the phrasing improvement is the paper's primary novelty; without it, the claim reduces to an intelligibility improvement (WER, MOSintelligibility) that could arise from other differences in the model architecture.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents ProsodyFM, a flow-matching TTS model that augments the MatchaTTS backbone with two prosody-oriented components: a Phrase Break Encoder that obtains break locations from a frozen PSST detector during training and uses a T5-based break predictor at inference, and a Terminal Intonation Encoder that represents last-word pitch shapes through a learnable bank of intonation tokens. The authors claim that ProsodyFM improves phrasing and intonation and thereby intelligibility relative to four SOTA TTS systems, and that it transfers better to unseen long and complex sentences and speakers. Evidence includes objective RMSEf0, WER, and F1break scores (Table 1), subjective MOS judgments (Table 2), an out-of-distribution comparison (Table 3), an ablation study (Table 4), and controllability case studies.","tokens_in":15511,"tokens_out":5120,"duration_ms":47594,"significance":"If the phrasing and intonation improvements are real, ProsodyFM would be a valuable contribution: it is an unsupervised, controllable prosody model with a relatively simple training recipe, and the paper's release of code, model checkpoints, and demo page supports reproducibility. The subjective MOS results, especially MOSbreak and MOSintonation, and the consistency of the ablations give some directional support. However, the objective phrasing evaluation is compromised by using PSST both to create training labels and to score synthesized breaks, which by construction favors ProsodyFM over baselines; the subjective evidence is based on only 15 utterances with labels derived from reference speech. The central claim that ProsodyFM effectively improves phrasing is therefore not yet established at the level claimed.","major_comments":[{"comment":"The F1break metric in Table 1 is not an independent test of phrasing. The same PSST system is used (i) as the frozen Phrase Break Detector that provides break locations during training (see 'Phrase Break Encoder'), (ii) as the source of ground-truth labels for fine-tuning the T5 Phrase Break Predictor (see 'Experimental Details' and Appendix B), and (iii) as the detector that produces phrase-break labels from both ground-truth and synthesized speech for the F1break computation. The four comparison systems were not trained against PSST labels, so ProsodyFM's higher F1break (62.76 vs 60.08 for MatchaTTS in Table 1) may reflect overfitting to PSST's boundary criteria rather than better phrasing. Please report an independent break evaluation, for example ToBI-style annotation on a held-out set or a second unsupervised break detector whose labels were not used in training, and state whether the F1break advantage survives.","section":"Objective Evaluation Metrics / Experimental Details"},{"comment":"The subjective phrasing and intonation scores are the main non-circular evidence for the headline claim, but the protocol is small and partly assumption-laden. The MOS test uses 15 utterances, labels derived from reference speech, and a non-parallel condition that assumes two sentences with similar semantics and structure share phrase-break and intonation labels (stated in 'Subjective Evaluation Metrics'); Appendix E says the labels were based on pitch contours plus perceptual judgments of two annotators, but no inter-annotator agreement or reliability statistic is reported. Please provide inter-annotator agreement, more test items, and a justification of the label-transfer assumption, or temper the phrasing and intonation claims accordingly.","section":"Subjective Evaluation Metrics"},{"comment":"The out-of-distribution claim is supported by a comparison against only MatchaTTS, and its headline metric F1break inherits the PSST label-source leakage. The statement that 'ProsodyFM in the out-of-distribution setting achieves matching performance with the four SOTA models in the in-distribution setting' compares across different test conditions and corpora; Table 3 reports no in-distribution row for the same VCTK-trained models, so this claim is not directly supported. Please add VCTK in-distribution results for both models, or at least clearly present matched in-distribution and out-of-distribution conditions, and use an independent break evaluation.","section":"Model Generalizability / Table 3"},{"comment":"The ablations show small objective differences (F1break 62.51 vs 61.25/61.06, WER 2.86% vs 3.13%/3.10%) with no significance tests, and the RMSEf0 differences are described as 'no substantial differences'. The conclusion that both the Phrase Break Encoder and Terminal Intonation Encoder are 'essential' for synthesizing highly intelligible speech is stronger than Table 4 supports. Please report confidence intervals or significance tests and calibrate the conclusion accordingly.","section":"Ablation Study / Table 4"}],"minor_comments":[{"comment":"There is a typo in Appendix A: 'During Predictor' should be 'Duration Predictor'.","section":"Appendix A"},{"comment":"GenerSpeech has no non-parallel MOS scores; please state this explicitly in the caption or main text rather than leaving the dash unexplained.","section":"Table 2"},{"comment":"The name 'w/o into break' is ambiguous; a clearer label such as 'w/o intonation and break' would improve readability.","section":"Ablation Study"},{"comment":"The claim that RMSEf0, F1break, and WER 'show a positive correlation' is not supported by a correlation statistic; please either compute it or soften the wording.","section":"Results"},{"comment":"Equation (1) should define T and state how DTW-mapped voiced frames are selected before the ratio is computed.","section":"Equation (1)"},{"comment":"Figure 1 is illustrative; the statement that the proposed smoothing 'alleviates pitch tracking errors' would benefit from a quantitative comparison of the smoothed contour against the other trackers.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The core issue is fixable: an independent break annotation study or a second break detector plus a larger human study would address the leakage. The paper is otherwise within scope and the method is potentially interesting; I would not reject, but the headline phrasing claim needs revision rather than the method being invalid."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know up front: this is a reasonably engineered TTS paper with a real evaluation flaw that sits right on the main claim. The objective phrasing metric, F1break, is computed with PSST—the same model that provides the break pseudo-labels during training. That makes the break score circular, and the advantage over baselines that never saw PSST is inflated by construction. The paper's own subjective MOS, though, is not circular: human raters heard better break and intonation similarity for ProsodyFM than for the four SOTA baselines, with 95% CIs and 21 raters per utterance. So the central claim survives, but with a smaller, more honest footprint than the abstract suggests.\n\nWhat is genuinely new: the Pitch Processor that subtracts a random offset to discard absolute pitch level, the learnable bank of intonation shape tokens trained directly with the OT-CFM loss, and the Phrase Break Encoder that turns PSST break detections into controllable break positions and durations. The authors release code and checkpoints, which is real. The ablation and the spectrogram case study show that the proposed modules do what they say: controlling break insertion/removal and rising/falling slopes in a visibly fine-grained way.\n\nSoft spots, in order of severity. First, the F1break circularity. It is not just a minor leak; it is the main objective evidence for the phrasing improvement. The MOSbreak scores are human and therefore independent, but they cover only 15 utterances, and the non-parallel condition relies on the assumption that semantically similar sentences share break and intonation labels—plausible but not established. Second, objective numbers in Tables 1, 3, and 4 come from a single run, with no error bars or significance tests. That matters for a systems paper claiming superiority. Third, the out-of-distribution claim is supported by a single baseline (MatchaTTS); calling that \"superior generalizability\" is overreaching. Fourth, Equation 9 for the text-pitch alignment loss is written with a negative L2 term. If actually used as written, it would push the embeddings apart, so it is likely a sign typo in the appendix—but it must be corrected.\n\nWho is this for: TTS researchers working on prosody control, and anyone who wants a reproducible system for break and intonation manipulation. It deserves a serious referee; the method is coherent, the code is out, and the subjective evidence is meaningful. I would send it to review with a request for an independent break annotation study and repeated-run statistics. The objective table as printed should be interpreted cautiously until then.","headline":"Solid TTS systems paper with a genuine evaluation leak in the objective phrasing metric; the human MOS evidence mostly carries the central claim, but the F1break numbers should not be trusted at face value.","tokens_in":15952,"tokens_out":1774,"would_cite":false,"duration_ms":18442,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ProsodyFM, a flow-matching TTS model, learns phrase breaks and pitch-shape tokens without prosodic labels and improves phrasing, intonation, and intelligibility.","keywords":["prosody","text-to-speech synthesis","flow matching","phrase break prediction","terminal intonation","pitch contour","intelligibility","unsupervised prosody modeling"],"falsifier":"Recompute the break F1 metric on the same synthesized utterances using an independently trained or human-annotated break labeler instead of the detector that also produced the training labels; if ProsodyFM's margin over the comparison models shrinks or reverses, the measured phrasing improvement is an artifact of the shared detector.","tokens_in":14893,"feed_emoji":"🎙️","tokens_out":11505,"duration_ms":100768,"temperature":0.7,"pith_summary":"ProsodyFM is a text-to-speech model built on a flow-matching backbone that tries to show that phrasing and terminal intonation can be learned without prosodic labels and that getting them right directly improves intelligibility. Its two supporting ideas are a phrase-break encoder paired with a duration predictor, which locate pauses and set their lengths, and a terminal-intonation encoder that learns a bank of pitch-shape tokens from a processed pitch contour instead of regressing absolute pitch values. On held-out speech, the authors report lower pitch error, lower word error rate, and higher break-detection F1 than four comparison models, and human listeners rated its breaks, terminal intonation, and intelligibility at or above those systems. In out-of-distribution tests on longer sentences and unseen speakers, the model degrades less than its backbone, and a case study demonstrates manual insertion or removal of breaks and continuous control of rising, falling, and level terminal tones.","feed_headline":"Prosody model outperforms four TTS systems on intelligibility","feed_subtitle":"The flow-matching TTS learns break placement and terminal pitch moves without prosodic labels, and humans rate it more intelligible.","key_machinery":"Two coupled modules carry the argument. The Phrase Break Encoder identifies phrase-final words, and the Duration Predictor assigns each break a speaker-conditioned duration, so placement and length of pauses are both controllable. The Terminal Intonation Encoder is built around a Pitch Processor that interpolates, smooths, and randomly offsets raw pitch contours to isolate their shape, and a bank of intonation shape tokens; the last-word pitch segment is compressed into a query that attends over the token bank, and the weighted sum becomes the terminal intonation embedding. At inference, an additional attention step aligns the reference speech's intonation patterns to the target text's phrase-final words, which is what lets the model transfer prosody even when the reference transcript differs from the text being synthesized.","core_discovery":"The paper's central claim is that a model which explicitly captures phrase-break locations, break durations, and terminal intonation shapes can make synthesized speech more intelligible than four current systems, even though it is trained with no explicit prosodic labels. The reported evidence is that ProsodyFM obtains the best or tied-best objective results on pitch error, word error rate, and break classification, and the highest human ratings for break similarity and terminal intonation similarity under both matched and mismatched reference-speech settings. The mechanism is unsupervised: speech is not annotated with break or intonation categories; instead, the model learns initial break locations from a frozen detector during training and from text at inference, adjusts break duration with a dedicated predictor, and represents terminal intonation as a weighted combination of learned shape tokens, with the pitch contour preprocessed to emphasize shape over absolute value.","pith_inferences":["An independent break-annotation study would provide a stronger check on the objective phrasing gain, since the detector used to create training labels is also used to score the output.","The intonation-shape token bank is a transferable design: it could be attached to other neural vocoders or acoustic models, and the same shape-over-value principle could be applied to prominence and rhythm rather than only terminal intonation.","The text-only phrase-break predictor could be used as a standalone front-end for other TTS systems, potentially improving their phrasing without retraining their speech backbones.","A natural next test is whether the learned shape tokens are speaker-independent; if they are, they could serve as a shared prosody prior for voice conversion or expressive speech editing."],"forward_implications":["Longer sentences with complex syntactic structure should be produced with pauses that respect phrase boundaries, reducing misparses and lowering word error rates.","End users get direct controls—inserting or deleting a break and changing the slope of a terminal rise or fall—without needing recorded reference prosody or labeled training data.","Because the model learns pitch shapes rather than absolute F0, its training should be less sensitive to pitch-tracker errors such as octave jumps and voicing misclassifications.","The out-of-distribution result suggests these prosody gains transfer to unseen speakers and sentence lengths, so the intelligibility improvement is not confined to the training distribution.","Human ratings of break and intonation similarity track intelligibility ratings, supporting the causal chain from improved phrasing and intonation to improved intelligibility."],"supporting_citations":[{"why":"Supplies the flow-matching backbone and the primary baseline whose architecture ProsodyFM extends.","marker":"Mehta et al. 2024"},{"why":"Supplies the frozen phrase-break detector that generates training labels and computes the objective break F1 metric.","marker":"Roll, Graham, and Todd 2023"},{"why":"Provides the optimal-transport conditional flow matching objective that trains the mel-spectrogram generator.","marker":"Lipman et al. 2023"},{"why":"Supplies the T5 model that is fine-tuned into the text-only phrase-break predictor used at inference.","marker":"Ni et al. 2022"},{"why":"Provides the style-token attention mechanism that the terminal intonation encoder adapts into a bank of intonation shape tokens.","marker":"Wang et al. 2018"},{"why":"Supplies the LibriTTS corpus used for training, validation, and testing in the main experiments.","marker":"Zen et al. 2019"},{"why":"Supplies the Whisper speech recognizer used to compute the word error rate intelligibility metric.","marker":"Radford et al. 2023"},{"why":"Supplies the HiFiGAN vocoder that converts the predicted mel-spectrograms into the audio heard in the evaluations.","marker":"Kong, Kim, and Bae 2020"}],"fun_headline_variants":["Unsupervised prosody control makes TTS speech more intelligible","Flow-matching TTS learns phrasing and intonation without labels","ProsodyFM: No labels, better phrasing and intonation than SOTA","TTS with unsupervised prosody control beats SOTA on intelligibility","Unsupervised phrasing and intonation control yields intelligible TTS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on the phrase-break detector being a trustworthy source of break labels, because that same detector is used to create the training labels, to fine-tune the text-only break predictor, and to compute the objective break-quality score.","fun_headline_variants_meta":{"raw":{"variants":["Unsupervised prosody control makes TTS speech more intelligible","Flow-matching TTS learns phrasing and intonation without labels","ProsodyFM: No labels, better phrasing and intonation than SOTA","TTS with unsupervised prosody control beats SOTA on intelligibility","Unsupervised phrasing and intonation control yields intelligible TTS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000608,"raw_usage":{"total_tokens":2847,"prompt_tokens":975,"completion_tokens":1872,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":1780}},"tokens_in":591,"tokens_out":1872,"duration_ms":12135,"temperature":1.0,"reasoning_tokens":1780,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:34:38.929902+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the break F1 metric on the same synthesized utterances using an independently trained or human-annotated break labeler instead of the detector that also produced the training labels; if ProsodyFM's margin over the comparison models shrinks or reverses, the measured phrasing improvement is an artifact of the shared detector.","supporting_citations":[{"cited_title":"J.; Jia, Y.; Chen, Z.; and Wu, Y","cited_arxiv_id":null,"evidence_quote":"Supplies the LibriTTS corpus used for training, validation, and testing in the main experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the flow-matching backbone and the primary baseline whose architecture ProsodyFM extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the frozen phrase-break detector that generates training labels and computes the objective break F1 metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the optimal-transport conditional flow matching objective that trains the mel-spectrogram generator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the style-token attention mechanism that the terminal intonation encoder adapts into a bank of intonation shape tokens."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HiFiGAN vocoder that converts the predicted mel-spectrograms into the audio heard in the evaluations."}],"review_version":1}