{"id":"68e0a24c-9806-4c45-ba10-aa0d0fb4afec","arxiv_id":"2501.04644","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-stage TTS framework that accepts arbitrary combinations of text, audio, and face prompts to control both style and timbre of generated speech.","lead":"FleSpeech is a speech synthesis system that lets users control a generated voice by combining different prompt types, such as a text description of the desired style, an audio sample for the speaker's timbre, or a face photo. It joins a language model and a flow-matching acoustic model so that style and timbre can be set separately, and it comes with a data pipeline for building the multimodal training set.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage-specific control is asserted but not tested: the pooled audio-prompt embedding contains both style and timbre, so in the central 'same timbre, different style' scenario the style prompt can be overridden.","rationale":"I keep the reader's CONDITIONAL verdict because the paper is coherent and the single-prompt comparisons are supportive, but the central stage-specific control claim has a load-bearing gap. The reader's weakest_assumption is that the pooled audio embedding captures all time-invariant information; my concern sharpens one consequence of that design: because the same pooled embedding also contains style, using it at the second stage for timbre control is not guaranteed to leave style alone. The reported multi-prompt tests deliberately use neutral audio prompts in the second stage, so they cannot detect style leakage. A 2x2 conflict test would settle whether the architecture actually achieves the 'same timbre, different style' scenario. I also note the internally trained face-attribute MLP in Appendix D.1 as a secondary evaluation concern, since it makes the Table 4 face-attribute accuracies partially circular, but the style/timbre leakage is more central to the framework's main claim. The concrete conflict test should be the decisive condition for acceptance.","tokens_in":17573,"tokens_out":9872,"duration_ms":99640,"concrete_test":"Run a 2x2 conflict experiment: fix one reference speaker's timbre and generate speech for all four cells (text emotion angry/neutral crossed with audio-reference emotion angry/neutral). Have 20 listeners rate perceived emotion of the outputs, or use an independent emotion classifier such as emotion2vec. Compute the effect of audio-reference emotion within each fixed text-emotion condition. If the audio-reference emotion significantly shifts perceived emotion (for example, more than 0.5 MOS or more than 15% classification accuracy) for a fixed text condition, the audio prompt leaks style and the stage-specific control claim fails. This directly tests the 'that woman's voice, but more angry' scenario that motivates the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's signature capability is stage-specific control: 'adjust style while preserving a selected speaker's timbre' means the text prompt at the language-model stage should set style while the audio prompt at the flow-matching stage should contribute only timbre. Section 3.3, however, states that the reference audio embedding 'captures all time-invariant information, such as style and timbre' and is obtained by average-pooling the output of the reference audio encoder. No disentanglement loss, attribute mask, or separate style/timbre encoding is described. The audio prompt therefore injects the reference speaker's style into the second stage, where it can override or blend with the style specified by the first-stage text prompt. The multi-prompt experiments in Section 6.2 do not expose this failure mode: 'we included emotional or neutral prompts in the first stage and only neutral prompts in the second stage,' so the audio prompt is never in conflict with the text style. The abstract's flagship examples—'Like this voice, but slower' and 'that woman's voice, but more angry'—are precisely the conflicting case. The claim that users can take one person's timbre and impose a different style therefore rests on an untested assumption about attribute disentanglement in the audio embedding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FleSpeech, a multi-stage speech generation framework in which a language model predicts semantic tokens from phonemes plus a global condition embedding, and a flow-matching module predicts acoustic latents from semantic tokens plus another condition embedding. A multimodal prompt encoder (MPE) maps text descriptions, reference audio, and face images into a unified embedding space using a query-based encoder and a diffusion objective, with the reference audio embedding as the training target. The model is trained in three stages on LibriHeavy plus a newly collected 616-hour multimodal dataset, and is evaluated on single-prompt controllable TTS, multi-prompt combinations, speaking style editing, and face-based voice conversion, using WER, speaker similarity, UTMOS, emotion/gender/attribute accuracy, and subjective MOS. The authors claim that FleSpeech supports any combination of prompts and enables stage-specific control, such as adjusting style while preserving a target speaker's timbre.","tokens_in":17853,"tokens_out":2756,"duration_ms":26597,"significance":"If the central claims hold, FleSpeech would be a meaningful step beyond single-prompt controllable TTS: it provides a concrete recipe for unify text, audio, and face prompts in one generative framework and demonstrates competitive objective and subjective results on several tasks. The paper also contributes a multimodal data collection pipeline and releases audio samples. Strengths of the evaluation include the use of external predictors (Whisper, WavLM, emotion2vec, UTMOS) for the main intelligibility, similarity, and quality metrics, subjective MOS with confidence intervals, and ablations on face captions and face embeddings. However, the signature capability of stage-specific control is asserted rather than demonstrated, because the experiments never place the text style prompt and the audio prompt in conflict, and at least one secondary evaluation metric is trained on the model's own embeddings. These gaps are load-bearing for the paper's flagship examples, so the manuscript needs additional evidence before the central claim can be accepted.","major_comments":[{"comment":"The stage-specific control claim is not tested in the reported experiments. Section 3.3 states that the reference audio embedding 'captures all time-invariant information, such as style and timbre' and is obtained by average-pooling the reference audio encoder output, with no disentanglement loss, attribute masking, or separate style/timbre encoders. In the multi-prompt experiments of Section 6.2, the second-stage audio prompt is always neutral or the first-stage text prompt is neutral, so the audio prompt never conflicts with the style specified by the text prompt. The abstract's headline scenarios—'Like this voice, but slower' and 'that woman's voice, but more angry'—are exactly the conflicting cases. The paper should either add experiments in which the text style prompt and the reference audio prompt specify different styles and report whether the generated speech follows the text style, or explicitly qualify the claim that FleSpeech can independently control style and timbre when both are specified in different stages.","section":"§3.3, §6.2, Abstract"},{"comment":"The face-attribute accuracy used to support the voice conversion claim is internally coupled to the proposed model. Appendix D.1 says that speaker embeddings are extracted from MPE and an MLP-based predictor is trained to infer facial attributes (BMI, age, race) from those embeddings. If the MLP is trained on the same MPE embedding space used to generate the speech, high accuracy can reflect the predictor learning the model's own biases rather than an independent face-to-voice correspondence. The paper should report how the MLP is trained and evaluated, use an external or human evaluation of the edited attributes, or otherwise demonstrate that the accuracy is not an artifact of the coupling. This is load-bearing for the 'voice conversion by face caption' result in Table 4.","section":"Appendix D.1, Table 4"},{"comment":"Objective results are reported without error bars, confidence intervals, or significance tests, while several baselines numbers are taken from the original papers or reproduced by the authors (marked ♢ and †). Differences between FleSpeech and baselines are often small (e.g., Text-prompt emotion accuracy 66.7 vs. 63.5, Audio-prompt WER 5.8 vs. 7.6), and the claims of 'significantly better' or 'optimal performance' are not supported by statistics. The authors should provide bootstrap confidence intervals or significance tests for the objective metrics, and report the number of test utterances used for each metric so the reader can assess the reliability of the comparisons.","section":"Tables 1-3"}],"minor_comments":[{"comment":"The header 'V olum' contains a typo and should read 'Volume'.","section":"Table 1"},{"comment":"'Comparsion' in the Section 6.1.1 heading and 'Langauge' in Section 3.2 are misspelled; please correct them.","section":"§1 and §6.1.1"},{"comment":"The description of masking says the audio modality prompt 'remains consistently present' while the main text says MPE accepts inputs from any modality and masks non-input modalities; please clarify whether audio is always required at training time and how inference with only text or face prompts is handled.","section":"Appendix B"},{"comment":"The similarity matrices in Figure 3 are discussed as supporting the superiority of the full FleSpeech model, but no quantitative comparison of the matrices is given; a scalar summary (e.g., mean off-diagonal vs. diagonal margin) would make the claim more concrete.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering contribution with a useful data pipeline, and the external metrics for the core TTS tasks are a strength. My main concern is that the paper's most distinctive capability—independent stage-specific control of style and timbre—is not actually exercised by the experimental design. The fix is feasible within the manuscript's scope: add conflicting-prompt experiments or meaningfully soften the claim. I also recommend requiring statistical support for the objective metric comparisons and clarification of the face-attribute evaluation. These are not grounds for rejection, but they require substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the architecture: a two-stage semantic-LM plus flow-matching generator, with a multimodal prompt encoder that maps text and face prompts into reference-audio embedding space via a diffusion objective. That combination is genuinely new, and the paper gives it a serious workout: 50k hours of training data, ablations on face captions, and a broad evaluation suite using external predictors (Whisper, WavLM, UTMOS, emotion2vec, human MOS). The data-collection pipeline, while not yet released, is a real contribution for a field that lacks multimodal TTS data. Credit where due: this is a solid systems paper with competitive numbers on single-prompt tasks.\n\nNow the soft spots, in proportion. The stress-test concern about stage-specific control is partly valid. The multi-prompt experiments in Section 6.2 deliberately pair emotional/neutral first-stage prompts with only neutral second-stage prompts, so they never put text style in direct conflict with audio style. That dodges the flagship \"same timbre, different style\" case. However, the speaking-style editing task in Section 6.3.1 does exactly that — text description for style, source audio for timbre — and FleSpeech preserves speaker similarity while getting 63.6% emotion accuracy, just below Audiobox. So the claim is not untested, but it is under-tested, and the paper should have included a direct conflicting-prompt experiment (e.g., angry text + calm reference) and reported whether the text style wins, loses, or blends.\n\nTwo smaller evaluation issues deserve referee attention. First, the face-attribute accuracy in Section D.1 uses an MLP trained on MPE speaker embeddings to predict BMI/age from those same embeddings; that is internally coupled and tells you little about true face-to-voice mapping. Second, objective metrics have no error bars or significance tests, and several baseline numbers are taken from other papers, which makes the comparisons hard to weigh. Code and data are promised only on acceptance, which is frustrating for a method that hinges on training details.\n\nThe central claim — flexible control across text, audio, and face prompts with stage-specific conditioning — is plausible and backed by enough independent evidence to take seriously. It deserves peer review, but the authors should be pushed to make the conflict test explicit, fix the circular face-attribute metric, and release the pipeline or at least the evaluation code. A clear revise-and-resubmit.","headline":"A capable controllable-TTS system paper whose headline flexibility claim is only indirectly tested; worth reviewing but needs a direct conflict-prompt experiment and cleaner internal metrics.","tokens_in":18357,"tokens_out":2166,"would_cite":true,"duration_ms":22888,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FleSpeech lets users control a generated voice with any mix of text, audio, and face prompts.","keywords":["controllable speech synthesis","multimodal prompt encoder","text-to-speech","voice conversion","speaking style editing","flow matching","face-driven speech synthesis","style transfer"],"falsifier":"Take a face image of a speaker whose voice has a distinctive accent or vocal quality not represented in the training labels, use it as the only prompt, and measure whether the synthesized voice matches that accent or quality in a forced-choice listening test. If listeners cannot identify the intended speaker from the generated voice, the face-to-embedding mapping is incomplete for attributes outside the annotated set.","tokens_in":17405,"feed_emoji":"🎙️","tokens_out":5600,"duration_ms":47309,"temperature":0.7,"pith_summary":"FleSpeech is a multi-stage speech generation framework that aims to make voice synthesis controllable by any combination of text, audio, and face prompts. The central claim is that because speech style and timbre are modeled at two different stages, a user can assign each attribute to a different prompt—for example, style from a text description and timbre from a reference recording. The paper proposes a multimodal prompt encoder that maps text and face representations into the space of reference audio embeddings, and a three-stage training strategy that keeps the generation backbone stable while learning these mappings. If the claim holds, users could request a voice like 'that woman's timbre, but angry' without recording anything new, and systems could generate a voice that plausibly matches a character's face.","feed_headline":"FleSpeech blends text, audio, and face prompts into one voice","feed_subtitle":"Style can come from a sentence, timbre from a recording, and the model keeps both.","key_machinery":"The multimodal prompt encoder (MPE) is the carrying mechanism. It is a query-based transformer, following the IP-Adapter design, that consumes embeddings from BERT-encoded text, CLIP-encoded face images, and a six-block reference audio encoder whose last block is average-pooled into a global style-timbre vector. The encoder runs a diffusion process that maps the prompt embeddings to the reference audio embedding space, and the resulting condition vector is fed separately into the language model stage (style-oriented) and the flow-matching stage (acoustic/timbre-oriented), with separate parameters for each stage.","core_discovery":"The paper's central claim is that speech synthesis can be flexibly controlled by arbitrary combinations of text, audio, and visual prompts, with each stage of a cascaded language-model-plus-flow-matching generator taking its own condition embedding. To make this work, FleSpeech trains a multimodal prompt encoder that uses learnable queries to extract speech-relevant information from each input modality, then runs a diffusion process that maps the query outputs to the same embedding space as a global, average-pooled reference audio embedding. The audio embedding serves as the anchor for all modalities: text and face prompts are only accepted insofar as they can be diffused into this space. The paper reports that this design lets the model separate style control (first stage) from timbre control (second stage), and shows objective and subjective results across single-prompt and multi-prompt conditions, plus extensions to speaking-style editing and voice conversion.","pith_inferences":["The two-stage design suggests a general recipe: any new speech attribute pair could be split across stages and conditioned by the same multimodal encoder, so the framework may scale to finer-grained control (e.g., rhythm in a third stage) without rearchitecting the generator.","Because text and face prompts are only defined through the reference audio embedding space, attributes that the audio prompt does not carry — like accent — should remain uncontrollable by text or face, a prediction consistent with the paper's stated limitation on face-derived information.","The released data collection pipeline could be reused to add new prompt modalities (e.g., gesture or scene captions) by generating descriptions and diffusing them into the same audio embedding space, effectively extending FleSpeech's input vocabulary at zero generator cost."],"forward_implications":["FleSpeech accepts prompts from any subset of modalities at inference, masking missing ones, so a user can go from text-only style control to full text+audio+face control in the same framework.","Style and timbre are modeled in separate stages, so a style text prompt can be combined with a reference audio prompt to adjust emotion while preserving a chosen speaker's voice.","Face prompts, when augmented with static and dynamic captions, generate speech whose timbre listeners match to the face, and can edit voice attributes such as age, BMI, and race at above 70% accuracy.","FleSpeech extends to speaking-style editing and voice conversion, modifying style or timbre from the same trained system without task-specific fine-tuning."],"supporting_citations":[{"why":"Supplies the query-based encoder design that lets textual and visual prompts be extracted into a unified condition vector.","marker":"Ye et al., 2023"},{"why":"Provides the reference audio encoder with average pooling that anchors the shared embedding space for all prompt modalities.","marker":"Min et al., 2021"},{"why":"Defines the static plus dynamic face-description strategy used to caption face prompts.","marker":"He et al., 2024"},{"why":"The multimodal-prompt baseline that FleSpeech extends and must outperform in comparisons.","marker":"Guan et al., 2024"},{"why":"Source of the emotional text-prompt corpus and of the SaLLE baseline used in evaluation.","marker":"Ji et al., 2024a"},{"why":"Introduces the language-model approach for semantic token prediction adopted as FleSpeech's first generation stage.","marker":"Wang et al., 2023"},{"why":"The LM-plus-flow-matching cascade architecture that FleSpeech's two-stage generator follows.","marker":"Du et al., 2024"},{"why":"The diffusion-transformer flow-matching backbone used for acoustic feature generation.","marker":"Esser et al., 2024"}],"fun_headline_variants":["Style from text, timbre from audio, both preserved","Mix text, audio, and face prompts for custom voice outputs","Choose your style with one prompt, voice with another","Control speech attributes independently via multimodal prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single average-pooled vector from a reference audio clip holds all the time-invariant style and timbre information a voice prompt needs, and that text or face prompts can be faithfully translated into that same vector space by the diffusion training.","fun_headline_variants_meta":{"raw":{"variants":["Style from text, timbre from audio, both preserved","Mix text, audio, and face prompts for custom voice outputs","Choose your style with one prompt, voice with another","Control speech attributes independently via multimodal prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000352,"raw_usage":{"total_tokens":1888,"prompt_tokens":885,"completion_tokens":1003,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":940}},"tokens_in":501,"tokens_out":1003,"duration_ms":9562,"temperature":1.0,"reasoning_tokens":940,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:27:28.691944+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a face image of a speaker whose voice has a distinctive accent or vocal quality not represented in the training labels, use it as the only prompt, and measure whether the synthesized voice matches that accent or quality in a forced-choice listening test. If listeners cannot identify the intended speaker from the generated voice, the face-to-embedding mapping is incomplete for attributes outside the annotated set.","supporting_citations":[],"review_version":1}