{"id":"73dc729a-05d8-48b0-936b-96d26bb14a7b","arxiv_id":"2412.18733","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Explicitly modeling intra- and inter-modal context interactions with four contrastive modules improves conversational prosody in TTS on DailyTalk.","lead":"This paper introduces III-CSS, a conversational speech synthesis system that uses four contrastive learning modules to model interactions between dialogue history and the next utterance across text and speech. It reports improved prosody and naturalness scores on the DailyTalk dataset, suggesting that explicitly separating intra- and inter-modal context helps conversational TTS.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation removes the contrastive losses, so the claimed benefit of contrastive interaction modeling is untested.","rationale":"The reader's weakest_assumption focuses on the hard ground-truth matrix in the contrastive learning objective. That is a plausible concern about the inductive bias, but it is secondary: even a flawed contrastive target could still provide useful training signal if the loss is part of the objective. The more load-bearing issue is that the contribution of the contrastive losses themselves is never isolated. The ablations systematically vary which modules are present, but never vary whether the contrastive loss is present while the module's feature extraction remains. This means the empirical evidence cannot distinguish 'contrastive interaction modeling helps' from 'extra cross-attended encoders help.' The paper also omits the total loss function, preventing readers or reviewers from knowing how the contrastive losses are balanced against the TTS loss or whether they are backpropagated to the synthesizer. This is a correctness risk in the central mechanism, not a stylistic complaint. I agree with the reader's conditional verdict because the reported gains are consistent but the missing specification and ablation leave the core claim underdetermined. My concern reinforces the need for a condition: the authors should provide the total training objective and an ablation that removes only the contrastive losses. I do not propose rejecting or accepting outright; the verdict remains conditional pending this evidence.","tokens_in":8873,"tokens_out":3764,"duration_ms":37162,"concrete_test":"Retrain the full I3-CSS model with the four contrastive losses (L_htnt, L_hsns, L_htns, L_hsnt) removed, i.e., set their loss weights to zero, while keeping all encoders, cross-attention interaction enhancement, and the Feature Aggregator addition identical. Evaluate the same metrics (N-DMOS, P-DMOS, MAE-P, MAE-E, MAE-D) on the same test split. If the results are statistically indistinguishable from the full model, the contrastive learning objective is not the source of the gains, and the central claim fails. Additionally, the authors should report the complete training objective and the weight for each contrastive loss to enable this ablation and to confirm that the contrastive gradients actually reach the synthesizer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that four contrastive learning-based interaction modules improve conversational prosody. The paper's ablations (Table II) remove individual modules or the cross-attention interaction enhancement (Abl.Exp.10), but every configuration that retains a module also retains its contrastive loss. There is no experiment that removes the contrastive losses while keeping the encoders and cross-attention features. Consequently, the reported gains could stem entirely from the added parameters and cross-attended feature extraction, with the contrastive objective contributing nothing. Compounding this, Section II-F does not specify the total training loss or the weighting coefficients for the four contrastive losses relative to the TTS reconstruction loss. Without this information, it is unclear whether the contrastive losses influence the synthesizer at all, or whether they only train the encoders and the resulting features are then frozen during synthesizer training. If the latter, the 'contrastive learning-based interaction enhancement' mechanism is not actually end-to-end, and the novelty claim reduces to adding extra cross-attended encoders, which is not new. This gap is load-bearing because the paper's contribution is explicitly the contrastive interaction modeling; if an ablation setting those losses to zero shows no performance drop, the central claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes I3-CSS, a conversational speech synthesis (CSS) system that explicitly models interactions between multimodal dialogue history (MDH) and the target utterance. The method introduces four contrastive-learning-based interaction modules covering historical-text/next-text, historical-speech/next-speech, historical-text/next-speech, and historical-speech/next-text combinations, whose learned features are added to the linguistic encoding before the acoustic decoder. Experiments on the DailyTalk dataset report that I3-CSS outperforms prior CSS baselines on naturalness DMOS, prosody DMOS, and MAE of pitch, energy, and duration. The authors also provide code and speech samples.","tokens_in":9055,"tokens_out":1941,"duration_ms":18704,"significance":"If the central claims are fully supported, the paper would make a useful contribution by being the first CSS system to explicitly decompose MDH-target interactions into intra-modal and inter-modal components and to train these components with contrastive objectives. The release of code and speech samples is a strength that facilitates reproducibility. However, the current evidence is incomplete: the training objective is not fully specified, the ablations do not isolate the contrastive losses, and the statistical reporting is insufficient to verify the claimed significance. These gaps currently prevent the paper from being accepted as-is, but they are addressable within the scope of the manuscript.","major_comments":[{"comment":"The manuscript never specifies the total training loss or how the four contrastive losses (L_cl_htnt, L_cl_hsns, L_cl_htns, L_cl_hsnt) are combined with the speech reconstruction loss. Section II-F only says that interaction modules are learned 'through a contrastive learning-based interaction enhancement mechanism' without giving weights or a total loss equation. It is therefore impossible to tell whether the contrastive losses influence the synthesizer at all, or whether they only train the encoders with features later frozen during synthesizer training. This is load-bearing because the paper's central contribution is the contrastive interaction modeling. Please state the full training objective and the joint-versus-frozen training procedure explicitly.","section":"II-F, II-B"},{"comment":"No ablation removes the contrastive losses while keeping the encoders and cross-attention interaction features. Abl.Exp.10 removes the 'interaction enhancement (IE)' mechanism, but IE is described as the cross-attention fusion, not as the contrastive objective. Consequently, the reported gains over Abl.Exp.1 could come entirely from the added parameters and cross-attended features, with the contrastive losses contributing nothing. The central claim that contrastive learning improves conversational prosody therefore remains untested. Please add an ablation that sets all contrastive loss weights to zero while retaining the four interaction modules and their cross-attention architecture.","section":"Table II"},{"comment":"The claim that I3-CSS 'significantly outperforms the baselines with a p-value less than 0.001' is not supported by the manuscript. No statistical test is named, no test statistic or confidence interval is reported, and the objective metrics (MAE-P, MAE-E, MAE-D) are reported as point values without error bars or significance tests. Also, the subjective evaluation uses 20 raters whose first language is not English, which may limit sensitivity for naturalness judgments. Please report the test used, the number of sentences and raters, and provide variance or confidence intervals for the objective metrics.","section":"Table I, III-B"},{"comment":"The contrastive ground-truth matrix M_gt assigns +1 only to the main diagonal and -1 to all off-diagonal entries, meaning that for every context length i, the accumulated history representation must be maximally similar to the immediately following utterance and equally dissimilar to all other utterances. Real conversations contain topic shifts, rhetorical questions, and non-adjacent dependencies, and this hard alignment may teach the interaction features to ignore relevant long-range context. The paper does not discuss or validate this modeling assumption. Please add a discussion or a small analysis (for example, an ablative variant with soft relevance labels or a comparison of diagonal versus off-diagonal similarity distributions) to justify this choice.","section":"II-B"}],"minor_comments":[{"comment":"The system is called 'I3-CSS' in the abstract and title but 'III-CSS' appears once in the abstract; please make the naming consistent throughout.","section":"Abstract / Title"},{"comment":"The text says 'In Abl.Exp.9, we remove the interaction enhancement mechanism,' but Table II labels this setting as Abl.Exp.10; the numbering should be corrected.","section":"Section III-E"},{"comment":"The notation 'F_t-intra_1→N−1' is used both for the full sequence and for the final accumulated feature; please disambiguate the sequence of prefix features from the final feature fed to the synthesizer.","section":"Section II-B"},{"comment":"The description of Abl.Exp.1 says 'removal of all intra-modal and inter-modal interaction modules,' but it is not clear whether the historical text and speech encoders are also removed or just not connected to the synthesizer; please clarify what remains in this configuration.","section":"Section III-C"},{"comment":"The motivating examples in the introduction are helpful, but they concern semantics and emotion rather than the specific prosodic features (pitch, energy, duration) measured later; a brief link between the examples and the objective metrics would improve readability.","section":"Section I"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid incremental CSS paper, not a breakthrough. The four-way decomposition of MDH–target interactions (text-text, speech-speech, text-speech, speech-text) is a reasonable and fairly new organizing scheme, and the paper reports consistent gains over DailyTalk, M2-CTTS, CONCSS, graph-based CSS, and ECSS on DailyTalk, with ablations showing each module helps. Code and samples are promised. That part is fine.\n\nWhere it gets soft: the central advertised mechanism is contrastive learning-based interaction enhancement, but no experiment removes the contrastive losses while keeping the encoders and cross-attention. The ablations remove whole modules, so they cannot separate the effect of the contrastive objective from the effect of the added parameters and cross-attended features. The stress-test note is right: if you zero out the contrastive losses and get no drop, the paper's novelty claim collapses to 'add cross-attended encoders,' which is not new. This is a load-bearing gap for a paper whose headline is contrastive interaction modeling.\n\nRelated: Section II-F never specifies the total training loss. Are the four contrastive losses combined with the TTS reconstruction loss in joint training, or are the interaction modules trained separately and then the features added to the synthesizer? The text says 'we input... and learn... through a contrastive learning-based interaction enhancement mechanism,' then the Feature Aggregator 'adds' the features. That reads like the interaction modules may be trained apart from the synthesizer, but it is not stated. Either way, the weighting of the four contrastive losses is absent. That is a reproducibility issue.\n\nOther smaller problems: the claimed p<0.001 for Table I has no described statistical test; objective metrics have no error bars or significance tests; the listening test is 20 non-native graduate students, which is small. The 'first CSS system to explore intra- and inter-modal interactions' overstates novelty relative to ECSS and related graph/contrastive work.\n\nOverall: the paper is worth a serious referee. The experiment is on one dataset, the gains are modest but consistent, and the missing contrastive-loss ablation and loss specification can be fixed in revision. If the contrastive losses turn out to be unnecessary, the paper becomes much weaker, but that is exactly what a referee should ask for. I would send it to peer review with a request for a full training-loss equation and a no-contrastive-loss ablation. If I were working in CSS, I'd cite it as related work despite the gaps, because the four-way taxonomy is useful.","headline":"Useful incremental CSS paper with a clear four-way interaction framing, but the contribution of the contrastive loss is untested because no ablation removes it, and the training objective is underspecified.","tokens_in":9536,"tokens_out":2552,"would_cite":true,"duration_ms":22038,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that explicitly modeling intra-modal and inter-modal interactions between the multimodal dialogue history and the target utterance improves conversational prosody, and supports the claim with a four-module contrastive…","keywords":["conversational speech synthesis","multimodal dialogue history","intra-modal interaction","inter-modal interaction","contrastive learning","conversational prosody","text-to-speech synthesis"],"falsifier":"Run the same system with the ground-truth similarity matrix replaced by soft relevance scores (from human annotation or a learned relevance predictor) and compare DMOS and MAE metrics; if prosody quality does not drop, the hard adjacency alignment is not the active mechanism behind the reported gains. A simpler check is to shuffle the history order at inference and see whether the prosody scores collapse.","tokens_in":8614,"feed_emoji":"🎙️","tokens_out":6823,"duration_ms":58035,"temperature":0.7,"pith_summary":"This paper tries to establish that conversational speech synthesis improves when the model explicitly learns two kinds of relation between the multimodal dialogue history and the next utterance: same-modality links (text-to-text and speech-to-speech) and cross-modality links (text-to-speech and speech-to-text). Earlier systems encoded the history independently or combined it with the target coarsely, leaving these complementary relations implicit. The proposed I3-CSS builds four interaction modules and trains each with a contrastive objective that aligns each accumulated history prefix with the representation of the immediately following utterance. On DailyTalk, the paper reports higher naturalness and prosody DMOS scores and lower pitch, energy, and duration errors than five compared baselines, with $p<0.001$.","feed_headline":"Modeling text–audio interactions improves conversational speech prosody","feed_subtitle":"Four contrastive modules link dialogue history to the next utterance, beating all five baselines on prosody.","key_machinery":"The load-bearing machinery is four contrastive interaction modules with shared encoder weights. Historical text is encoded with Sentence-BERT plus speaker embeddings, historical speech with IEMOCAP-fine-tuned Wav2Vec 2.0 features, and each module aggregates prefix-level features through cross-attention. Training aligns each accumulated history representation with the immediate next utterance's representation by minimizing the mean squared error between a cosine-similarity prediction matrix and a ground-truth matrix of 1s and $-1$s; at inference only the history is available, so the trained modules supply the four interaction features that condition the FastSpeech2-style synthesizer and HiFi-GAN vocoder.","core_discovery":"The central claim, stated in the paper's own terms, is that separately modeling intra-modal and inter-modal context interactions between the multimodal dialogue history (MDH) and the target utterance lets a CSS system infer both the semantics and the prosody of the target from what came before, producing speech with more appropriate conversational prosody. Each of the four trained modules corresponds to one text/speech pairing, and their features are added into the target utterance's linguistic encodings before the acoustic decoder. The ablation results are used to argue that every module contributes and that the interaction-enhancement mechanism matters: removing all modules drops N-DMOS to 3.598 and P-DMOS to 3.615, while removing only the enhancement mechanism drops the full system from 3.864/3.876 to 3.752/3.747.","pith_inferences":["A natural extension not explored in the paper is replacing the hard 1/-1 alignment with soft relevance weights; that would test whether treating every adjacent pair as maximally related is too rigid for topic-shifting dialogues.","The same four-way contrastive design could be reused as conditioning for other sequence-generation tasks where a multimodal history shapes an upcoming output, such as emotion recognition in conversation or expressive voice cloning.","A cheap diagnostic of whether the gains come from dialogue structure rather than speaker/style leakage is to shuffle the history order at inference; if prosody scores stay high, temporal alignment is not what the model exploits."],"forward_implications":["Explicitly decomposing history-target relations into four modality pairings gives a measurable prosody and naturalness gain over treating history as an independent context or fusing it coarsely with the target.","Each pairing carries non-redundant information: any single module improves over none, any two modules improve over any single one, and the full set is best.","Because inference needs only the dialogue history, the trained interaction modules can be used at synthesis time before the target utterance is spoken, without access to its recorded audio.","The contrastive alignment objective turns the notion of 'appropriate next prosody' into a concrete training signal, and the paper's numbers tie that signal to improved DMOS and lower acoustic prediction error."],"supporting_citations":[{"why":"Supplies the DailyTalk dataset used for training and evaluation.","marker":"[5]"},{"why":"Multimodal multiscale context encoder baseline that I3-CSS is compared against.","marker":"[6]"},{"why":"Contrastive-based context comprehension baseline that I3-CSS extends contrastive history modeling from.","marker":"[7]"},{"why":"Graph-based dialogue history baseline providing a comparative reference.","marker":"[10]"},{"why":"Heterogeneous graph emotion-aware baseline that combines target and history, the closest prior approach.","marker":"[11]"},{"why":"Sentence-BERT provides the pre-trained text encoder for semantic features of historical and next text.","marker":"[18]"},{"why":"Wav2Vec 2.0 provides pretrained speech representations fine-tuned for prosodic features.","marker":"[19]"},{"why":"FastSpeech2 supplies the TTS encoder, length regulator, and variance adapter used by the synthesizer.","marker":"[22]"},{"why":"HiFi-GAN acts as the vocoder that renders the final conversational speech.","marker":"[23]"}],"fun_headline_variants":["New model weaves text and audio context for better speech prosody","Contrastive modules capture cross-modal context for natural prosody","Intra- and inter-modal cues boost conversational speech style","Dialogue history text and speech jointly shape next utterance prosody"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each accumulated history prefix should be made maximally similar to the immediately following utterance and equally dissimilar to every other utterance, even though real dialogues have softer, shifting, and sometimes non-adjacent relevance.","fun_headline_variants_meta":{"raw":{"variants":["New model weaves text and audio context for better speech prosody","Contrastive modules capture cross-modal context for natural prosody","Intra- and inter-modal cues boost conversational speech style","Dialogue history text and speech jointly shape next utterance prosody"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000149,"raw_usage":{"total_tokens":1203,"prompt_tokens":962,"completion_tokens":241,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":170}},"tokens_in":578,"tokens_out":241,"duration_ms":2703,"temperature":1.0,"reasoning_tokens":170,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:31:12.719132+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same system with the ground-truth similarity matrix replaced by soft relevance scores (from human annotation or a learned relevance predictor) and compare DMOS and MAE metrics; if prosody quality does not drop, the hard adjacency alignment is not the active mechanism behind the reported gains. A simpler check is to shuffle the history order at inference and see whether the prosody scores collapse.","supporting_citations":[{"cited_title":"Dailytalk: Spoken dialogue dataset for conversational text-to-speech,","cited_arxiv_id":null,"evidence_quote":"Supplies the DailyTalk dataset used for training and evaluation."},{"cited_title":"M 2-ctts: End-to-end multi-scale multi-modal conversational text-to- speech synthesis,","cited_arxiv_id":null,"evidence_quote":"Multimodal multiscale context encoder baseline that I3-CSS is compared against."},{"cited_title":"Concss: Contrastive-based context comprehension for dialogue- appropriate prosody in conversational speech synthesis,","cited_arxiv_id":null,"evidence_quote":"Contrastive-based context comprehension baseline that I3-CSS extends contrastive history modeling from."},{"cited_title":"Inferring speaking styles from multi-modal conversational context by multi-scale relational graph convolutional networks,","cited_arxiv_id":null,"evidence_quote":"Graph-based dialogue history baseline providing a comparative reference."},{"cited_title":"Emotion rendering for conversational speech synthesis with heterogeneous graph-based con- text modeling,","cited_arxiv_id":null,"evidence_quote":"Heterogeneous graph emotion-aware baseline that combines target and history, the closest prior approach."},{"cited_title":"Sentence-BERT: Sentence embeddings using Siamese BERT-networks,","cited_arxiv_id":null,"evidence_quote":"Sentence-BERT provides the pre-trained text encoder for semantic features of historical and next text."},{"cited_title":"Fastspeech 2: Fast and high-quality end-to-end text to speech,","cited_arxiv_id":null,"evidence_quote":"FastSpeech2 supplies the TTS encoder, length regulator, and variance adapter used by the synthesizer."}],"review_version":1}