{"id":"122a1717-b4af-4f49-b917-7870d526cb0b","arxiv_id":"2411.13314","paper_version":4,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"I2TTS is an end-to-end TTS that conditions on CLIP image features and a frozen reverberation classifier to synthesize scene-matched, speaker-adaptive speech, reporting gains on SRE, MOS, and WER.","lead":"A new text-to-speech system, I2TTS, generates speech whose reverberation matches a given scene image, aiming to create more immersive audio for virtual reality and gaming. The paper reports large gains on scene-matching metrics, but the main metric is never defined and may be biased by the model's own training classifier.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SRE metric is undefined and may use the same frozen classifier used as a training loss, so the headline spatial-matching improvement is unverified.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the SRE metric is undefined, and the only described reverb classifier (SRC) is used as a training loss. I agree that this is the central vulnerability because SRE is the main objective evidence for the paper's core claim of scene-aware reverberation matching. The paper's own text supports this concern: SRC is trained on the same convolutionally reverberated speech and then frozen for computing training loss, while SRE is never given a formula or evaluation protocol. If the same classifier is used to score SRE, the reported improvement is at least partly tautological. The subjective IMOS result is too small and unpaired to independently substantiate spatial perception, so the claim rests on SRE. I do not see a stronger internal inconsistency in the architecture itself; the VITS-based end-to-end design and the zero-shot speaker encoder are plausible. The appropriate response is to require the authors to specify SRE and verify it with an independent classifier. Since the reader already recommended REJECT, my read does not change that verdict. I assign agreement_with_reader = 'agree' because the reader's weakest assumption and my concern coincide.","tokens_in":8459,"tokens_out":3750,"duration_ms":43037,"concrete_test":"Obtain the SRE evaluation details from the authors or the demo release and determine whether the frozen SRC is used for scoring. Independently, train a new reverberation-classifier (e.g., on Image2Reverb RIRs held out from SRC training or on real room impulse responses) and recompute Table I SRE for ViT-TTS, MS2KU-VTTS, and the proposed model. If the 37-point gap shrinks below a few points, the reported SRE was largely an artifact of optimizing the same classifier; if the gap persists, the spatial-matching claim survives this specific attack.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of spatially accurate, scene-matched speech rests primarily on the reported drop in SRE from 64.2% (ViT-TTS) to 27.2% (proposed). However, SRE is never formally defined: Section IV-B says only that 'SRE measures the correctness of the room acoustics of the generated speech,' and the metric is inconsistently named 'Space Recognition Error' in IV-B but 'Scene Reverb Error' in Table II. Meanwhile, Section IV-A states that the Speech Reverberation Classifier (SRC) is pretrained on the same convolved speech, frozen, and used only to compute the classification loss of generated speech. If SRE is computed with this same SRC—or with a classifier substantially derived from it—then the model is directly optimizing the evaluation metric during training. The improvement over ViT-TTS would then reflect overfitting to the SRC's decision boundary and would not establish generalizable acoustic-scene matching. The paper does not describe the SRE computation pipeline, does not report the classifier used for evaluation, and releases no code, so this circularity cannot be ruled out. Because IMOS is the only other scene-matching metric and is based on 20 listeners rating 6 samples without pairing, it cannot independently carry the spatial-perception claim. The architectural contributions may be sound, but the central evidential support is unverifiable as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents I2TTS, an end-to-end multispeaker text-to-speech system that takes a text input, a visual scene prompt, and an optional speaker reference to synthesize reverberant speech. The architecture combines a CLIP-based image encoder with an adapter, a speaker encoder for zero-shot voice cloning, and a frozen Speech Reverberation Classifier (SRC) used during training to refine acoustic-scene alignment, all built on a VITS backbone. Evaluations on LJSpeech, VCTK, and Image2Reverb report improved word error rate, mel cepstral distortion, speaker similarity, and especially a large reduction in Space Recognition Error (SRE) compared with ViT-TTS and MS2KU-VTTS, alongside subjective NMOS/SMOS/IMOS scores.","tokens_in":8718,"tokens_out":3262,"duration_ms":36732,"significance":"If the reported results hold, the paper makes a useful contribution to multi-modal TTS by integrating visual scene conditioning directly into an end-to-end VITS-based pipeline and by demonstrating zero-shot speaker adaptation. The architectural direction is reasonable and the ablation variants provide some evidence for the usefulness of the CLIP encoder and the SRC module. However, the central claim of spatially accurate scene-reverb matching rests on an undefined metric and a potentially circular evaluation protocol. The paper also provides no statistical significance measures for its objective metrics and only a very small subjective evaluation. These issues currently prevent the claimed superiority from being accepted, although they are addressable in a revision.","major_comments":[{"comment":"SRE is never formally defined. The text only states that 'SRE measures the correctness of the room acoustics of the generated speech,' and Table II inconsistently calls it 'Scene Reverb Error.' The paper does not specify the classifier used for evaluation, the set of reverb classes, how ground-truth labels are assigned, or how errors are aggregated. Since the reported drop from 64.2% (ViT-TTS) to 27.2% (Proposed) is the main objective evidence for the spatial-matching claim, the metric must be defined precisely for the results to be verifiable.","section":"§IV-B and Table II"},{"comment":"The evaluation of SRE risks being circular. The SRC is pretrained on the same convolved speech data used for training the TTS system, is frozen during TTS training, and is used to compute a classification loss on generated speech. If SRE is computed with this same SRC or a substantially similar classifier, then the model is directly optimizing the evaluation metric, and the improvement over baselines would reflect overfitting to the SRC's decision boundary rather than generalizable acoustic-scene matching. The paper must either use an independent held-out classifier for SRE evaluation or explicitly report the relationship between the training-time classifier and the evaluation classifier.","section":"§IV-A and §III-C"},{"comment":"Objective metrics are reported without error bars, confidence intervals, or statistical significance tests. For example, WER differences between the proposed method (7.6) and the w/o CLIP ablation (7.8) are small, and SECS differences (0.62 vs. 0.57) may be within normal variation. Without repeated runs or significance testing, the claim of 'consistently superior performance' across objective metrics is not supported.","section":"§IV-C, Table I"},{"comment":"The subjective IMOS evaluation is described as engaging 20 listeners each evaluating 6 test samples, but the paper does not state whether each listener rated all systems on the same samples, how many judgments per condition were collected, or whether the reported standard deviations (<0.25) are across listeners or across samples. This is too small a basis to support the strong IMOS claims in Table I and Table II, especially the environment-wise IMOS differences of 0.15-0.3.","section":"§IV-B"},{"comment":"The role of the SRC is ambiguous. The method section says the SRC 'refines' the mel-spectrogram and 'adjusts' reverberation, but the only described use is computing a cross-entropy loss on the predictions of the generated speech. No forward-pass post-processing or refinement equation is given. The contribution summary in §I also alternately calls it a 'Scene Relevance Classifier' and a 'Speech Reverberation Classifier,' which makes the intended mechanism unclear. Please clarify whether the SRC acts only as a training-time regularizer or performs any inference-time correction.","section":"§III-C"}],"minor_comments":[{"comment":"There is a typo in 'reveives the mel-spectrogram' that should be 'receives.'","section":"§III-C"},{"comment":"The metric is named 'Space Recognition Error' in §IV-B but 'Scene Reverb Error' in Table II; please use one consistent name.","section":"§IV-B"},{"comment":"The description of the baseline is ambiguous: 'the VITS backbone trained on the clean speech followed by convolution with RIR' could mean the generated clean speech is convolved at inference time, but it is unclear whether the baseline sees any scene information. Please specify.","section":"§IV-A"},{"comment":"The Grad-CAM visualization in Figure 2 is qualitative and does not demonstrate that the highlighted image regions are acoustically relevant. Consider quantifying the correlation between attention maps and acoustic parameters such as RT60.","section":"§IV-E"},{"comment":"No code or demo audio samples are provided in the manuscript; releasing at least the evaluation code for SRE computation would greatly help reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The reader's rejection is understandable given the undefined and potentially circular SRE metric. I recommend major revision rather than rejection because the central deficiency is a missing definition and evaluation protocol, which can be fixed by re-running the evaluation with an independent held-out classifier and reporting the computation details. However, if the authors cannot provide such an evaluation, the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read this one with moderate expectations and came away with a clear bottom line: the architecture is a reasonable incremental step, but the paper's central claim is unverified because the main metric is undefined and possibly circular.\n\nWhat's actually new: I2TTS builds on VITS and adds a CLIP-based image encoder with an MLP adapter, a speaker encoder for zero-shot voice adaptation, and a frozen reverberation classifier (SRC) used as an auxiliary training loss. That combination—end-to-end, no external vocoder—is a fair advance over ViT-TTS and MS2KU-VTTS, which rely on a separate vocoder. The ablations (w/o SRC, w/o CLIP) are helpful, and the paper is clearly written with relevant citations.\n\nThe problems are in the evaluation. SRE is never defined. It appears as 'Space Recognition Error' in Section IV-B and 'Scene Reverb Error' in Table II. The SRC is pretrained on the same convolved speech, frozen, and used to compute a classification loss on generated speech. If SRE uses that same SRC—or a classifier derived from it—then the headline improvement (64.2% to 27.2%) is largely tautological, because the model directly optimizes the evaluation metric. The paper gives no description of the SRE computation pipeline and releases no code, so the circularity cannot be ruled out. The only other scene-matching metric, IMOS, is based on 20 listeners rating 6 samples without pairing, so it can't independently support the claim. The word 'refinement' for the SRC is also misleading: it's a frozen loss, not a post-hoc refinement module.\n\nIf the authors define an independent evaluation metric (e.g., a held-out classifier not used in training), report error bars, and release code, this could become a useful contribution to scene-aware TTS. As written, the evaluation does not support the central claim, and I would not cite it. Still, the architectural idea is sound enough that I'd send it to peer review rather than desk-reject—the fix is tractable, and a serious referee should see it.\n\nRecommendation: major revision required; reject in current form.","headline":"Plausible incremental architecture, but the central spatial-matching claim is unverified because SRE is undefined and may be computed with the same frozen classifier used as a training loss.","tokens_in":9273,"tokens_out":2851,"would_cite":false,"duration_ms":28622,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"I2TTS is an end-to-end multimodal TTS framework that synthesizes speech whose reverberation matches a visual scene prompt, while preserving naturalness and supporting zero-shot voice adaptation.","keywords":["text-to-speech","spatial audio","reverberation","visual scene prompt","multimodal TTS","zero-shot speaker adaptation","CLIP","VITS"],"falsifier":"Compute the paper's Space Recognition Error using a room classifier trained on held-out room impulse responses that were never mixed into the training set, and compare the same generated samples against ViT-TTS; if the 27.2%-versus-64.2% gap shrinks or disappears, the spatial-matching claim is not established. A complementary check is a forced-choice listening test where raters pick which of several room photos matches each generated audio clip.","tokens_in":8270,"feed_emoji":"🎙️","tokens_out":7261,"duration_ms":72326,"temperature":0.7,"pith_summary":"This paper claims that text-to-speech can be made spatially aware by generating reverberant speech directly from a text input plus a photo of the acoustic scene, rather than applying room acoustics as a post-processing step. The proposed system, I2TTS, feeds CLIP visual features through an adapter into a VITS backbone, uses a frozen speech-reverberation classifier as a training-time refinement loss, and adds a speaker encoder for zero-shot voice adaptation. The authors report that this end-to-end design beats the earlier image-conditioned systems ViT-TTS and MS2KU-VTTS on word error rate, spectral distance, speaker similarity, scene recognition error, and listener ratings, while keeping naturalness intact. A sympathetic reader would take the central claim to be that scene-aware reverberation can be learned jointly with content and speaker identity without degrading intelligibility or voice quality.","feed_headline":"Now text-to-speech can match a room's acoustics from a photo","feed_subtitle":"End-to-end model fuses CLIP scene features and speaker identity to generate reverberant speech without a separate vocoder.","key_machinery":"The load-bearing machinery is a VITS text-to-speech backbone that receives two extra conditionings: a scene embedding obtained by passing the scene image through a CLIP image encoder and an MLP adapter, and a speaker embedding obtained from a short reference clip with a StyleEncoder-style module. A Speech Reverberation Classifier (SRC), made of four 2-D convolution blocks plus a transformer block and a classification head, is pretrained on the same convolved training speech, frozen, and used to compute a cross-entropy loss on the generated mel-spectrogram so the decoder learns to produce reverberation consistent with the image. The adapter is also guided with a cross-entropy loss against the reverb category, tying the visual embedding to the acoustic label space.","core_discovery":"The central claim is that I2TTS synthesizes high-quality immersive speech from text and visual scene prompts in one end-to-end pipeline, producing speech whose reverberation matches the image's room acoustics. Built on a VITS backbone, the model injects a CLIP-derived scene embedding and a speaker embedding into the generative flow, and a frozen Speech Reverberation Classifier refines the generated mel-spectrogram by predicting the reverb class and feeding a cross-entropy loss back during training. In the reported evaluations the full model reaches a 27.2% space recognition error against 64.2% for ViT-TTS and 57.1% for MS2KU-VTTS, with the best WER (7.6%), MCD (4.22), SECS (0.62), and IMOS (3.96), which the authors take as evidence that scene and spatial matching can be achieved without compromising naturalness.","pith_inferences":["Extension: the same scene-embedding adapter could be paired with different acoustic-transformation targets, such as head-related transfer functions for binaural direction, making spatial TTS extend from room reverb to full three-dimensional audio.","Extension: because the frozen classifier used to train the model was pretrained on the same convolved audio that forms the training data, the reported SRE gains should be re-checked with a classifier trained on held-out room impulse responses or with human scene-matching tests.","Extension: feeding the SRC with the ground-truth reference speech as a contrastive negative could turn the refinement stage into a true alignment mechanism rather than a soft label predictor."],"forward_implications":["Because the decoder outputs the waveform directly, the method avoids the external vocoder that previous image-conditioned TTS systems required, removing a source of synthesis artifacts.","A scene photo becomes a practical control signal: the same text can be rendered with the acoustics of a cathedral, a hallway, or a recording booth from the corresponding image.","The speaker encoder allows an unseen voice to be placed into a specified acoustic scene from a few seconds of reference speech, enabling personalized VR and gaming audio.","The SRC refinement appears to help in both wide and narrow environments, with reported SRE of 25.3% and 28.6% respectively, suggesting the scene conditioning does not overfit to one acoustic scale."],"supporting_citations":[{"why":"Supplies the VITS backbone that I2TTS modifies for end-to-end waveform generation.","marker":"[3]"},{"why":"Supplies the zero-shot speaker adaptation approach and the style-encoder design used in the speaker encoder.","marker":"[5]"},{"why":"The visual-conditioned TTS baseline whose scene-reverb matching and quality are compared against the proposed method.","marker":"[12]"},{"why":"The multi-source spatial-knowledge TTS baseline that is compared in the experiments.","marker":"[13]"},{"why":"Provides the CLIP image encoder whose visual features are adapted into acoustic scene embeddings.","marker":"[14]"},{"why":"Provides the Image2Reverb dataset of paired images and room impulse responses used to create the reverberant training and ground-truth speech.","marker":"[24]"},{"why":"Provides the StyleEncoder architecture adopted to extract speaker identity from reference speech.","marker":"[27]"},{"why":"Defines the mel cepstral distortion metric used to measure spectral fidelity of the synthesized speech.","marker":"[31]"},{"why":"Defines the speaker encoder cosine similarity metric used to assess speaker identity preservation.","marker":"[32]"}],"fun_headline_variants":["Image prompts give text-to-speech room-aware reverb","TTS uses a photo to synthesize room-matched acoustics","Visual scene prompts drive spatial audio in speech synthesis","From image to immersive speech: I2TTS matches room reverb","New TTS model reads a photo to set the room's echo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The main evidence that the synthesized speech matches the scene rests on a Space Recognition Error metric that the paper never defines, while the classifier used to train the model is pretrained on the same convolved speech, so the reported drop in SRE could partly reflect the model learning to fool that classifier.","fun_headline_variants_meta":{"raw":{"variants":["Image prompts give text-to-speech room-aware reverb","TTS uses a photo to synthesize room-matched acoustics","Visual scene prompts drive spatial audio in speech synthesis","From image to immersive speech: I2TTS matches room reverb","New TTS model reads a photo to set the room's echo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1542,"prompt_tokens":956,"completion_tokens":586,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":502}},"tokens_in":572,"tokens_out":586,"duration_ms":5952,"temperature":1.0,"reasoning_tokens":502,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:33:18.428554+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the paper's Space Recognition Error using a room classifier trained on held-out room impulse responses that were never mixed into the training set, and compare the same generated samples against ViT-TTS; if the 27.2%-versus-64.2% gap shrinks or disappears, the spatial-matching claim is not established. A complementary check is a forced-choice listening test where raters pick which of several room photos matches each generated audio clip.","supporting_citations":[{"cited_title":"Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,","cited_arxiv_id":null,"evidence_quote":"Supplies the VITS backbone that I2TTS modifies for end-to-end waveform generation."},{"cited_title":"Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,","cited_arxiv_id":null,"evidence_quote":"Supplies the zero-shot speaker adaptation approach and the style-encoder design used in the speaker encoder."},{"cited_title":"Multi-source spatial knowledge understanding for immersive visual text-to-speech,","cited_arxiv_id":null,"evidence_quote":"The multi-source spatial-knowledge TTS baseline that is compared in the experiments."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP image encoder whose visual features are adapted into acoustic scene embeddings."},{"cited_title":"Image2reverb: Cross-modal reverb impulse response synthesis,","cited_arxiv_id":null,"evidence_quote":"Provides the Image2Reverb dataset of paired images and room impulse responses used to create the reverberant training and ground-truth speech."},{"cited_title":"Mel-cepstral distance measure for objective speech quality assessment,","cited_arxiv_id":null,"evidence_quote":"Defines the mel cepstral distortion metric used to measure spectral fidelity of the synthesized speech."}],"review_version":1}