{"id":"63314dd5-3860-4a8c-b47c-22f517a48d1e","arxiv_id":"2507.00808","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Multi-turn textual directions can iteratively refine the speaking style of synthesized speech through a learned embedding refiner, with modest but measurable alignment to the directions.","lead":"This paper builds a text-to-speech system that takes multiple rounds of textual directions, like a voice director coaching an actor, to refine synthesized speech step by step. It also creates a recorded dataset of direction-performance cycles and shows listeners can hear intended style changes, though the refinements are still weaker than a real director's recordings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Style-refiner transfer from recorded to synthetic embeddings is unquantified; the iterative claim rests on a small subjective sample.","rationale":"The paper makes a modest claim (refinement 'to some extent') and includes an honest discussion of limitations (§5.1–5.2), so I do not see grounds for rejection. The reader's conditional verdict is appropriate. The most load-bearing technical risk is the train/inference mismatch: a refiner regressing embeddings of recorded speech is used to edit embeddings of synthetic speech, and no analysis of that shift is reported. The evaluation does use synthetic prompts, which is independent support, but the sample is small and the scores are near the 'overall alignment only' anchor, so the domain gap remains the point where the central claim is least secure. My concrete test would quantify the shift and directly connect it to refinement quality. Since the reader already flagged this as the weakest assumption and reached CONDITIONAL, my read does not change the verdict.","tokens_in":11278,"tokens_out":7750,"duration_ms":91896,"concrete_test":"Use the held-out test direction cycles to compare the style refiner's behavior on recorded versus synthetic pre-refined speech. For each cycle, extract the input embedding x from the recorded actor speech (Recorded-N) and from the TTS output for the same text/style (Identical-0 or Guided-N); compute the L1 distance between the refiner's predicted x' and the target x' from the actor's next take under both inputs, and measure the distribution distance (e.g., MMD or Fréchet distance) between recorded and synthetic x in the 384-d space. If the synthetic-prompt loss is substantially larger, or the distributions are far apart, the recorded-to-synthetic transfer assumption fails and the iterative refinement claim needs stronger evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim—that iterative refinement follows user directions—requires that the style refiner trained in §2.3 on embeddings of recorded human speech transfer to embeddings of the TTS model's own synthetic speech inside an iterative loop. Training uses L1 regression between x and x' extracted from recorded actor takes; inference feeds x extracted from Identical-0, Guided-N, or the previous Iterative output and passes the predicted x' back through STL into the TTS. The manuscript does not measure or correct this recorded-to-synthetic embedding shift. The subjective tests in §4.2–4.3 partially exercise the shift because Table 3 uses synthetic prompts, but on only 16 direction cycles (§4.2) and 50 sessions (§4.3), with mean scores around the 'overall alignment only' anchor; Table 4 shows that position-specific and linguistic directions already fail. If the domain gap is large, the refiner may exploit spurious differences between recorded and synthetic embeddings, making success on global style labels insufficient evidence for multi-step refinement across the full dataset. This is a correctness risk, not a mere consensus disagreement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-interaction TTS framework in which a style refiner takes the speech embedding extracted from the current synthesis plus a textual direction, predicts a refined embedding, and feeds it back through a style-token layer into a FastSpeech2-based backbone TTS. To support training, the authors collected a Japanese dataset of 1,606 direction cycles with two voice actors, 52 speaking styles, and LLM-generated sequences of two or three iterative directions, and augmented the direction texts with LLM-based paraphrases. Three crowdsourced subjective evaluations are reported: an iterative style-refinement MOS test comparing the proposed iterative condition with an actor-guided oracle and an identical-speech baseline, a style-refinement accuracy test using matched versus random directions, and a naturalness MOS test. The paper concludes that the method achieves iterative style refinement that follows user directions 'to some extent' without degrading naturalness, and it candidly discusses remaining failures for position-specific and linguistic instructions.","tokens_in":11441,"tokens_out":6490,"duration_ms":79844,"significance":"If the central claim holds, this is a useful step toward interactive TTS: the dataset of direction cycles is a potential resource for the community, the style-refiner design is simple and modular, and the idea of modeling the voice-director/voice-actor loop is well motivated. The authors deserve credit for including an oracle condition, a random-direction control, confidence intervals, significance testing, and a publicly available demo page, as well as for explicitly acknowledging in Sect. 5 which direction types are not captured. However, the observed effect is modest: the overall iterative score (2.90) is below the 'Overall alignment only' anchor of 3 and well below the oracle (3.27), and the evidence for the multi-step claim rests on a small number of tested cycles and on an unquantified assumption that a refiner trained on recorded-speech embeddings transfers to synthetic-speech embeddings inside an iterative loop.","major_comments":[{"comment":"The style refiner is trained exclusively on embeddings extracted from recorded speech (x from the pre-refined take, x' from the refined take), yet at inference, including in the Iterative condition of Table 3, it is applied to embeddings extracted from synthetic speech (Identical-0, Iterative-N). The paper does not quantify the distribution shift between recorded and synthetic embeddings, nor does it include any domain-adaptation or correction. This is load-bearing for the central 'multi-interaction' claim: if the synthetic embeddings lie outside the training distribution, the L1 regression may rely on spurious recorded/synthetic differences, and errors could accumulate over iterations. The subjective test in Sect. 4.2 partially exercises the shift, but with only 16 direction cycles total. Please add a quantitative analysis of the recorded-to-synthetic embedding gap (for example, centroid distances or nearest-neighbor overlap for matched content and style), and/or report iterative refinement results on a substantially larger set of cycles, including session-by-session scores.","section":"Section 2.3 / Section 4.2"},{"comment":"The only non-oracle baseline in the iterative-refinement evaluation is Identical (no refinement), so the significant improvement over Identical demonstrates that the refiner changes something, not that the change follows the direction. The absolute scores are not strong: the overall Iterative mean is 2.90, which is below the 'Overall alignment only' anchor of 3, and the Actor-Guided oracle itself reaches only 3.27. To support the abstract's claim that refinement is 'in accordance with users' directions', please add a baseline that receives the same direction text without the speech prompt (e.g., PromptTTS-style conditioning) or a single-step version of the proposed refiner, and report effect sizes and direction-type-wise results. As it stands, the comparison to Identical is a minimal sanity check rather than a demonstration that the iterative loop is the source of the improvement.","section":"Section 4.2"},{"comment":"The style-refinement accuracy experiment is conducted under the Single-shot condition of Table 2, where the pre-refined and refined inputs are recorded speech or synthetic speech generated directly from recorded prompts; it therefore does not measure behavior inside the iterative loop, which is the central new claim. The paper also states that Random (Similar) 'yield comparable performance to Matched' without reporting a significance test for that pair. In addition, Table 4 and Sect. 5.1 show that position-specific and linguistic directions are not handled by the current global-embedding refiner. The abstract and conclusion should either explicitly restrict the claim to global paralinguistic style directions or report results separately by direction category; as written, the claim is broader than the evidence.","section":"Section 4.3 / Table 4"}],"minor_comments":[{"comment":"The sentence 'STL post-process x was applied' appears to be missing a verb form and should read 'STL post-processes x'.","section":"Section 2.2"},{"comment":"The statement 'As the style refiner, we utilized Gemma2 LLM fine-tuned on a Japanese dataset' is inconsistent with the architecture in Sect. 2.3, which describes cross-attention, an aggregation module, and FiLM. Please clarify whether Gemma2 is used only as the direction-text encoder or is the full refiner.","section":"Section 3.2"},{"comment":"The 'n.s.' markers in Figures 4 and 6 are not tied to specific pairs; please state which comparisons are not significant (for example, Iterative versus Actor-Guided within the Joy and Surprise groups, or the cycle-by-cycle comparisons).","section":"Figures 4 and 6"},{"comment":"The split of Random into Similar and Dissimilar relies on ChatGPT o3-mini similarity scores with a threshold, but no reliability or agreement measure is reported for this automatic classification; a brief validation would strengthen the interpretation.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"This is a solid workshop-level contribution with a novel dataset and honest reporting, but for a journal the evidence for the headline iterative-refinement claim is thin. The two issues I would prioritize are (1) a direct quantitative characterization of the recorded-to-synthetic embedding shift, and (2) a stronger baseline and effect-size reporting that isolate the contribution of the iterative loop. If those are addressed, I would be willing to reconsider."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper does something genuinely new—textual multi-turn style refinement for TTS, with a purpose-built direction-cycle dataset—but the evidence is suggestive, not conclusive, and the strongest claim (iteration tracks user directions) rests on a small subjective sample with a modest effect size.\n\nWhat's new: PromptTTS and VoiceCraft are single-turn; this is the first multi-direction refinement loop I've seen. The dataset construction is thoughtful: directions generated by LLM but recorded with future directions concealed from the actor, and the refiner is trained on embeddings from actual recorded takes. The paper is also pleasantly candid about limitations—Section 5.1 admits global embeddings can't handle word-specific instructions, Section 5.2 notes no linguistic change capability, and the conclusion explicitly says 'to some extent.' That honesty earns credit.\n\nSoft spots: there is no experimental comparison against any text-prompt TTS baseline, so the incremental value of the iterative loop over a single-turn prompt is not directly quantified. The refiner is speaker-dependent, and the recorded-to-synthetic embedding shift is never measured. The subjective refinement scores hover around the 'overall alignment only' anchor (2.9 vs 3.27 oracle on a 5-point scale), and the significant differences are mostly against an 'Identical' control rather than against a strong alternative. The Random-condition split into similar/dissimilar was done post-hoc with a ChatGPT score, which is a weak point. No code, data, or weights are released, though demo audio exists. These are real but proportionate concerns—none are fatal to the idea.\n\nThe stress-test worry about the domain gap is fair but partially mitigated: the evaluation in §4.2 uses synthetic reference speech, so the loop is exercised, just on a small sample (16 cycles, 50 sessions). It's a correctness risk worth flagging in any review, not a reason to desk-reject.\n\nBottom line: this is a serious first step on a useful task, with honest reporting and a reproducible-enough experimental protocol for a workshop. I'd send it to peer review and hope a strong reviewer pushes for a baseline comparison and a domain-gap analysis. I'd bring it to reading group; I'd cite it if I worked on expressive TTS.","headline":"Genuinely new multi-turn TTS refinement task with honest reporting, but the evidence is suggestive, not conclusive, and the domain-gap risk deserves scrutiny.","tokens_in":11992,"tokens_out":1997,"would_cite":true,"duration_ms":21252,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text-to-speech accepts follow-up directions to refine style.","keywords":["text-to-speech","expressive speech synthesis","speaking style refinement","iterative interaction","style embedding","voice direction dataset","text-prompted TTS","multi-step refinement"],"falsifier":"Starting from a recorded take, apply the style refiner, take the synthetic output, apply a second direction, and measure whether each step moves the embedding toward the corresponding refined recorded take and whether the subjective directional score holds. If the second-step movement shrinks sharply or the score degrades compared with the first step, the recorded-to-synthetic transfer assumption breaks and the iterative loop would not survive multiple rounds.","tokens_in":11067,"feed_emoji":"🎙️","tokens_out":6663,"duration_ms":74185,"temperature":0.7,"pith_summary":"This paper proposes a text-to-speech system that treats the synthesizer as a voice actor and lets a user act as a voice director, issuing several written directions in sequence. The authors build a new dataset that records actors responding to two or three staged direction rounds, then train a style refiner that moves the speech embedding of the previous output toward the requested direction. Subjective tests show that iterating directions moves the synthesized style in the requested direction, that this tracks the oracle condition which uses real recordings, and that naturalness is not degraded. If the claim holds, expressive TTS moves from re-prompting from scratch to a loop where the previous synthesis itself becomes the input to the next refinement.","feed_headline":"Text-to-speech accepts follow-up directions to refine style","feed_subtitle":"A new dataset and style refiner let users adjust synthesized speech in a sequence without losing naturalness.","key_machinery":"The load-bearing mechanism is the style refiner trained separately from the backbone TTS model. The backbone is conditioned on an embedding from a self-supervised speech encoder and a style-token layer; the refiner takes the intermediate frame-level representation $r$, attends to the direction text, aggregates the result, and applies a FiLM conditioning layer to convert the current embedding $x$ into the refined embedding $x'$. Training uses L1 regression on embeddings extracted from recordings: the input is the pre-refined take, the target is the actor's refined take, and the direction text is the conditioning. At inference this learned map is applied repeatedly, feeding each synthesized utterance back as the next iteration's input.","core_discovery":"The central discovery is that iterative style refinement can be implemented by manipulating a single global speech embedding, with no change to the linguistic content. Given a pre-refined embedding $x$ and a direction text, a style refiner predicts a refined embedding $x'$, which the backbone TTS model renders as speech; at the next round that synthetic speech supplies the new pre-refined embedding. Trained on a dataset of direction cycles, recorded actor performances with two or three textually generated directions per manuscript, the refiner yields synthesized speech whose style follows the directions, although scores remain below the oracle condition in which the real actor take is used. Naturalness ratings show no significant difference among single-shot, iterative, and oracle conditions. The authors frame the result as a first step: directions that target specific words or linguistic features such as pauses are not yet realized, because a global embedding cannot alter individual positions in a sentence.","pith_inferences":["Beyond the paper: the practical ceiling is set by the unseen loop condition; the refiner is trained on recorded pre- and post-refinement embeddings but deployed on the TTS model's own synthetic embeddings, so multi-round performance depends on how far synthetic embeddings drift from recorded ones.","A direct stress test would be to train on pairs where both sides are synthetic, or to add an embedding-space alignment loss, and then compare iterative refinement quality; the paper does not report such a comparison.","Because the refiner is speaker-dependent, the natural next boundary is cross-speaker generalization; without speaker conditioning or retraining, the interaction loop may not transfer to unseen voices.","The dataset ties each direction cycle to a style label, so it could be repurposed for automatic evaluation, for example by measuring whether the embedding trajectory across sessions moves toward the embedding of the labeled target style."],"forward_implications":["Refinement becomes a sequence: a user can issue a direction, hear the result, and then issue another direction, with each new synthesis feeding the next round.","Iterative direction-following does not cost naturalness: subjective naturalness was statistically indistinguishable between single-shot, iterative, and oracle-generated speech.","The method handles complex and abstract directions beyond simple high/low or fast/slow prompts, because the direction texts were created to mimic real voice-directing language.","Style-group breakdowns show the effect across fear, joy, anger, sadness, and surprise, with joy and surprise matching the oracle statistically in refinement score.","Directions that name a particular word, phrase, or pause location are explicitly outside the reach of a global-embedding approach, pointing toward fine-grained TTS as the next step."],"supporting_citations":[{"why":"Supplies the style-token layer that lets the backbone TTS be conditioned on a speech embedding.","marker":"[18]"},{"why":"Represents the text-prompt TTS approach this work extends and contrasts with, whose prompts are simpler than the direction texts.","marker":"[22]"},{"why":"Provides the speaking-style taxonomy and manuscript alignment used to construct the interactive dataset.","marker":"[26]"},{"why":"Supplies the self-supervised speech encoder whose representations become the embeddings the style refiner manipulates.","marker":"[27]"},{"why":"Introduces the FiLM conditioning layer used to convert the current embedding into the refined embedding.","marker":"[30]"},{"why":"Provides the Easy, Hard, and Medium prompt templates used to augment direction texts for robustness.","marker":"[34]"},{"why":"Defines the FastSpeech 2 backbone architecture that renders the refined embedding as speech.","marker":"[35]"},{"why":"Supplies the HiFi-GAN vocoder that produces the final waveform in the speech-generation pipeline.","marker":"[37]"}],"fun_headline_variants":["TTS lets users refine style in a feedback loop","Multi-step TTS: direct voice style after synthesis","Interactive TTS: refine speech style with follow-up text","Iterative style refinement for text-to-speech","Voice directors? TTS accepts style tweaks mid-course"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The style refiner is trained on recordings by real actors, but at use it is applied to the TTS system's own synthetic speech; the method assumes synthetic speech sits close enough to the recorded training examples that the same refinement still works across multiple rounds.","fun_headline_variants_meta":{"raw":{"variants":["TTS lets users refine style in a feedback loop","Multi-step TTS: direct voice style after synthesis","Interactive TTS: refine speech style with follow-up text","Iterative style refinement for text-to-speech","Voice directors? TTS accepts style tweaks mid-course"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000298,"raw_usage":{"total_tokens":1688,"prompt_tokens":873,"completion_tokens":815,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":738}},"tokens_in":489,"tokens_out":815,"duration_ms":8306,"temperature":1.0,"reasoning_tokens":738,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:06:18.838248+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Starting from a recorded take, apply the style refiner, take the synthetic output, apply a second direction, and measure whether each step moves the embedding toward the corresponding refined recorded take and whether the subjective directional score holds. If the second-step movement shrinks sharply or the score degrades compared with the first step, the recorded-to-synthetic transfer assumption breaks and the iterative loop would not survive multiple rounds.","supporting_citations":[{"cited_title":"From page to stage: The director’s interpretation and picturization of a script,","cited_arxiv_id":null,"evidence_quote":"Supplies the style-token layer that lets the backbone TTS be conditioned on a speech embedding."},{"cited_title":"Photorealistic text-to- image diffusion models with deep language understanding,","cited_arxiv_id":null,"evidence_quote":"Represents the text-prompt TTS approach this work extends and contrasts with, whose prompts are simpler than the direction texts."},{"cited_title":"PaLM: Scaling language modeling with path- ways,","cited_arxiv_id":null,"evidence_quote":"Provides the speaking-style taxonomy and manuscript alignment used to construct the interactive dataset."},{"cited_title":"V oice puppetry: Exploring dramatic performance to develop speech synthesis,","cited_arxiv_id":null,"evidence_quote":"Introduces the FiLM conditioning layer used to convert the current embedding into the refined embedding."},{"cited_title":"Fine- grained robust prosody transfer for single-speaker neural text-to- speech,","cited_arxiv_id":null,"evidence_quote":"Provides the Easy, Hard, and Medium prompt templates used to augment direction texts for robustness."},{"cited_title":"Daft- Exprt: Cross-speaker prosody transfer on any text for expressive speech synthesis,","cited_arxiv_id":null,"evidence_quote":"Defines the FastSpeech 2 backbone architecture that renders the refined embedding as speech."}],"review_version":1}