{"id":"07483e30-2ea9-4969-a7b1-2e75bd3007da","arxiv_id":"2412.20048","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CrossSpeech++ disentangles language and speaker information in speech generation and reports improved cross-lingual TTS naturalness over prior systems on four languages.","lead":"This paper presents CrossSpeech++, a speech synthesis system that splits text-to-speech generation into language and speaker parts to create speech in a new language while keeping a speaker's voice. The authors report higher naturalness than several baselines on English, Chinese, Japanese, and Korean, though the gains are modest and code is not released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Disentanglement claim rests on unvalidated speaker-independence of MMS features; Table IV's w/oLA SECS increase suggests residual speaker leakage in LDG.","rationale":"The reader's weakest assumption is on target: the SSL/perturbation pipeline is the only mechanism enforcing speaker removal in the LDG, and the paper does not validate that the last hidden layer of MMS after perturbation is actually speaker-independent. I strengthen this with direct evidence from Table IV: removing the linguistic adaptor increases SECS in both cross-lingual and intra-lingual settings, and the authors explicitly attribute this to residual speaker information entangled in the text representations. That admission undercuts the core disentanglement claim. The t-SNE visualization in Fig. 6(a) is suggestive but not a rigorous test of speaker leakage. A quantitative speaker-probe experiment would settle whether the LDG is truly speaker-independent. The evaluation weaknesses noted by the reader (overlapping MOS confidence intervals, no error bars on objective metrics, small margin over CrossSpeech) are real but secondary; the more load-bearing issue is whether the proposed mechanism actually achieves disentanglement. The reader's CONDITIONAL verdict remains appropriate, so no change is needed.","tokens_in":17015,"tokens_out":5793,"duration_ms":63127,"concrete_test":"Extract LD features (output of the LD decoder, before the final conv projection) for all test utterances from the released or author-provided checkpoint. Train a logistic-regression speaker classifier on a subset of speakers and evaluate on held-out speakers. Compare classification accuracy for the full CrossSpeech++ model and the w/oLA ablation. If the full model's LD features are classified above chance, or above the w/oLA model, the LDG retains speaker information, contradicting the disentanglement claim. If both are at chance, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CrossSpeech++ 'effectively disentangles language and speaker information' depends on the LDG output being free of speaker-dependent information. The only mechanism enforcing this is the choice of target for the linguistic adaptor (Sec. IV-C): the last hidden layer of MMS extracted from perturbed waveforms. The paper does not directly measure speaker information in the LD features; the t-SNE in Fig. 6(a) is qualitative, and t-SNE can hide clustering. More importantly, the paper's own ablation data (Table IV) are in tension with the claim: removing the linguistic adaptor (w/oLA) increases SECS in both cross-lingual (0.772 vs. 0.761) and intra-lingual (0.783 vs. 0.777) settings, and the authors attribute this to 'residual speaker information entangled within the text representations' (Sec. VII-C). If the LDG still encodes residual speaker identity, the architectural separation does not achieve the stated disentanglement, and the reported gains could come from the extra variance predictors and auxiliary SSL targets rather than from a speaker-independent language pathway. Since no quantitative speaker-leakage metric is reported, the central mechanism is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CrossSpeech++, a non-autoregressive cross-lingual TTS system built on FastPitch with conformer blocks. The architecture splits generation into a Language-dependent Generator (LDG) and a Speaker-dependent Generator (SDG), whose outputs are summed to obtain the mel-spectrogram. The LDG uses Mix-Dynamic Speaker Layer Normalization (MDSLN), a language-dependent variance (LDV) adaptor, and a linguistic adaptor that regresses SSL features extracted from perturbed waveforms using a frozen MMS model; the SDG uses Dynamic Speaker Layer Normalization (DSLN) and a speaker-dependent variance (SDV) adaptor. Experiments cover four languages with 22 speakers and compare against FastPitch variants, CrossSpeech, and zero-shot models, reporting MOS, SMOS, UTMOS, SECS, and CER. The central claim is that this decoupling effectively disentangles language and speaker information in the output acoustic feature space, yielding significant cross-lingual quality improvements.","tokens_in":17166,"tokens_out":4628,"duration_ms":49029,"significance":"If the claims are substantiated, the contribution is a practical architectural recipe for cross-lingual TTS that moves disentanglement from input representations into the output acoustic feature space, building on the authors' prior CrossSpeech. The paper includes a useful set of ablations, an online aligner that avoids per-language forced alignment, and a comparison across several MMS layers; the demo page and use of public corpora and component models substantially support reproducibility of the system itself. The evaluation is broader than in many TTS papers (MOS, SMOS, UTMOS, SECS, CER, plus ablations), which is a strength. However, the statistical support for the headline claims and the quantitative evidence for the disentanglement mechanism are weaker than the abstract's language suggests, and the zero-shot comparison is not a matched experimental protocol.","major_comments":[{"comment":"The abstract claims 'significant improvements' and 'outperforming existing methods by a large margin,' but the numbers in Table II do not consistently support this. Cross-lingual MOS for CrossSpeech++ is 4.06 ± 0.09 vs. 3.93 ± 0.08 for CrossSpeech, which at face value is a 0.13 difference with overlapping 95% confidence intervals and no significance test; SMOS is actually lower (3.82 vs. 3.87) and SECS is lower (0.761 vs. 0.776). Objective metrics are reported as point estimates with no error bars or significance tests. Please provide significance testing for MOS/SMOS (e.g., pairwise tests with multiple-comparison correction) and error estimates for UTMOS, SECS, and CER, or temper the wording of the claims so that they do not overstate the evidence.","section":"Table II, Abstract"},{"comment":"The central disentanglement claim rests on the assertion that the LDG output is essentially free of speaker information, but the only direct evidence is the t-SNE in Fig. 6(a), which is qualitative and can hide clustering structure. The paper's own ablation in Table IV shows that removing the linguistic adaptor (w/oLA) raises SECS in both cross-lingual (0.772 vs. 0.761) and intra-lingual (0.783 vs. 0.777) conditions, and Sec. VII-C attributes this to 'residual speaker information entangled within the text representations.' This is an explicit acknowledgment that the language pathway still carries speaker-dependent information. Please add a quantitative speaker-leakage metric computed directly on LD features (e.g., speaker classification accuracy, or the average cosine similarity of LD features from the same speaker vs. different speakers) and report it for the full model and for ablations. Without such a measurement, the claim that the architecture 'effectively disentangles' language and speaker is not established.","section":"Sec. IV-C, Fig. 6, Table IV"},{"comment":"The comparison with VALL-E X and XTTS-v2 is not a matched apples-to-apples evaluation. CrossSpeech++ is trained and evaluated on utterances from the same 22-speaker corpus, including the test speakers (the 80/10/10 split in Sec. VI-A is within-speaker), whereas VALL-E X and XTTS-v2 are zero-shot systems that have not been adapted to the target speakers. In addition, Table V excludes Korean because VALL-E X does not support it, changing the test set relative to the main evaluation. This protocol difference biases the comparison in favor of CrossSpeech++. Please either (a) retrain or fine-tune CrossSpeech++ in a strict zero-shot protocol with reference utterances from unseen speakers, or (b) clearly relabel the comparison as 'trained speaker-dependent vs. zero-shot' and add a speaker-dependent baseline such as adapted FastPitch so that the effect of the proposed modules can be isolated.","section":"Sec. VII-E, Table V"},{"comment":"The cross-lingual and intra-lingual conditions are not directly comparable because cross-lingual evaluation uses only four representative speakers per language while intra-lingual uses all speakers. This is visible in the results: for every system, UTMOS is higher in the cross-lingual condition than in the intra-lingual condition (e.g., CrossSpeech++ 3.791 vs. 3.343; CrossSpeech 3.279 vs. 3.039), which is surprising given the usual finding that cross-lingual synthesis is harder. Please clarify the test-set composition (number of utterances per speaker, speaker overlap across languages) and explain this pattern. If the metric is sensitive to speaker identity or language, report per-language and per-condition results so that the reader can assess whether the cross-lingual gains are driven by a small set of favorable speakers.","section":"Sec. VII-A, Table II"}],"minor_comments":[{"comment":"The caption contains the typo 'architecutre'; please correct it.","section":"Fig. 2 caption"},{"comment":"In Eq. (5), the index i-1 at i=1 references an out-of-range value; please specify how the sequence is padded (e.g., p_0 defined as the first value or treated as equal).","section":"Eq. (5)"},{"comment":"Binarizing pitch and energy discards the magnitude of rises and falls; a sentence of justification or a reference showing that binary contours suffice for the intended prosody modeling would help the reader evaluate this design choice.","section":"Sec. IV-B"},{"comment":"The row for Ground Truth reports only intra-lingual scores; please also report cross-lingual ground-truth scores (or state that these are unavailable), as the current format makes it hard to judge the absolute gap to natural speech.","section":"Table II"},{"comment":"Please report the number of utterances used in the subjective tests (the text says 50 clips, but it is unclear whether that is the total or per condition) and the inter-rater agreement, so that the confidence intervals can be interpreted.","section":"Sec. VI-E"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely of interest to the speech-synthesis community and the architecture is plausible, but the evaluation and the disentanglement evidence need substantial strengthening before the claims can be accepted. The referee report focuses on issues that are fixable within the manuscript's scope: add significance tests or temper the claims, provide a quantitative speaker-leakage measure, and revisit the zero-shot comparison protocol. The authors should also consider disclosing that the '22 speakers' setting is small and that the test speakers were seen in training, since this affects how readers interpret the cross-lingual generalization claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: CrossSpeech++ is a sensible follow-up to the authors' own CrossSpeech, not the large-margin breakthrough the abstract promises. The genuinely new pieces—MDSLN, binary pitch/energy targets, and the SSL linguistic adaptor trained on perturbed MMS features—are reasonable engineering choices, and the ablations actually do support that each component contributes. Credit where due: the ablation study is thorough, the evaluation includes both subjective MOS with confidence intervals and multiple objective metrics, and the use of information perturbation to strip speaker-dependent content from SSL targets is a nice idea.\n\nThe soft spots are real but not disqualifying. The 'significant improvements' and 'large margin' language in the abstract is not backed by Table II: cross-lingual MOS goes from 3.93 to 4.06, SMOS drops from 3.87 to 3.82, and SECS drops from 0.776 to 0.761. Objective metrics come without error bars or significance tests, and the corpus is only 22 speakers. The zero-shot comparison is also partial, since VALL-E-X doesn't support Korean. The deeper concern, which the stress-test note gets right, is that the disentanglement claim rests on an unvalidated assumption: that MMS layer 24 features from perturbed waveforms are free of speaker information. The paper never directly measures speaker leakage in the LD features; the t-SNE in Fig. 6(a) is qualitative and can hide clustering. The ablation results in Table IV actually point the same way: removing the linguistic adaptor raises SECS (0.772 vs 0.761 cross-lingual, 0.783 vs 0.777 intra-lingual), which the authors attribute to residual speaker information in text representations. That is an honest admission, but it undercuts the clean 'language-dependent generator is speaker-free' story. The gains in naturalness and intelligibility are still real, but they may come more from the extra variance predictors and auxiliary loss than from true architectural disentanglement.\n\nBottom line: this is a coherent engineering paper, clearly written, with honest limitations discussed (e.g., the low-resource caveat). It deserves a serious referee, not a desk reject, but the authors should be pushed to add error bars or significance tests on objective metrics, provide a quantitative speaker-leakage measure for the LD features, and tone down the abstract. I'd bring it to a reading group only if someone in the group works on TTS or disentanglement; otherwise it's a bit niche. I wouldn't cite it in my own work, but I'd send it out for review.","headline":"Competent incremental extension of CrossSpeech with solid ablations, but the headline claims outrun the evidence and the disentanglement story is only indirectly supported.","tokens_in":17805,"tokens_out":2014,"would_cite":false,"duration_ms":22904,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CrossSpeech++ claims that splitting speech generation into language- and speaker-dependent branches, trained with self-supervised linguistic targets, yields more natural cross-lingual speech.","keywords":["cross-lingual speech synthesis","language-speaker disentanglement","text-to-speech","self-supervised speech representations","speaker generalization","prosody modeling","multilingual TTS","Mix Dynamic Speaker Layer Normalization"],"falsifier":"A direct test is to train a speaker classifier on the language-dependent features produced by the LDG for a held-out set of speakers and languages: if it can predict speaker identity above chance, the LD branch has not actually removed speaker information. A complementary check is to swap only the language embedding while holding the speaker fixed and measure speaker-embedding cosine similarity of the output; a large drop in similarity would indicate speaker leakage into the language branch.","tokens_in":16733,"feed_emoji":"🗣️","tokens_out":11982,"duration_ms":98708,"temperature":0.7,"pith_summary":"CrossSpeech++ tries to close the gap between cross-lingual text-to-speech (speaking a language the target speaker never used in training) and ordinary same-language text-to-speech by attacking the language-speaker entanglement problem at the output stage rather than only at the input stage. It splits speech generation into a language-dependent generator, which produces pronunciation and intonation patterns free of speaker bias, and a speaker-dependent generator, which colors those patterns with timbre and pitch. The two branches are trained together so that their summed mel-spectrogram matches the target, with extra losses steering each branch toward only its own kind of information. On a four-language corpus of monolingual speakers, the paper reports large gains in naturalness and intelligibility over prior systems, including its own earlier CrossSpeech model, in both cross-lingual and same-language settings. If these results hold, separating generation this way could make high-quality multilingual voices practical from ordinary monolingual data.","feed_headline":"Language/speaker split lifts cross-lingual speech synthesis","feed_subtitle":"Two separate generators produce more natural speech in four languages from monolingual training data.","key_machinery":"The central object is the additive decomposition of the mel-spectrogram into language-dependent and speaker-dependent representations, each produced by its own branch of a FastPitch-style non-autoregressive TTS model. The load-bearing components are MDSLN, a MixStyle-inspired layer normalization that perturbs the LD encoder with statistics from randomly shuffled speaker embeddings (mixing weight drawn from Beta(2,2)); the LDV adaptor, which turns ground-truth pitch and energy into binary up/down sequences per text token so the language branch learns direction of intonation but not absolute pitch; the linguistic adaptor, which trains an LD decoder to predict the last-hidden-layer features of MMS computed on information-perturbed waveforms (formant shifting, pitch randomization, frequency shaping) and reinforces them with a CTC text predictor; and, in the SDG, DSLN plus the SDV adaptor, which predict continuous pitch and energy to restore speaker-specific prosody. The argument is that by making the language branch predict only speaker-shifted, binary, and perturbed targets, speaker information cannot be encoded in it, while the speaker branch must carry the residual identity.","core_discovery":"On the paper's own terms, the central discovery is that disentangling language and speaker in the output acoustic feature space, by assigning each to a separate generator, is more effective than disentangling only in the input token space. The language-dependent generator is trained to be speaker-generalizable through Mix Dynamic Speaker Layer Normalization, which mixes randomly shuffled speaker statistics during training, plus a language-dependent variance adaptor that predicts binary pitch and energy directions and a linguistic adaptor that regresses self-supervised features extracted from a perturbed waveform by a frozen wav2vec2.0 model (MMS, last hidden layer). The speaker-dependent generator uses Dynamic Speaker Layer Normalization and a speaker-dependent variance adaptor to add timbre and pitch contours. The final mel-spectrogram is the sum of the two branches' outputs, and the full model is trained with a multi-term loss including mel reconstruction, duration, LD and SD variance, linguistic feature L1, and CTC text prediction. The paper reports that this scheme achieves the best naturalness scores (MOS and UTMOS) and the lowest character error rate (CER) among compared cross-lingual systems, with only a small dip in speaker-similarity scores relative to CrossSpeech, which it attributes to more accurate text-driven pronunciation.","pith_inferences":["If the output-space disentanglement is real, the same two-branch additive design could be applied to voice conversion or accent transfer, where one wants to keep linguistic content fixed while swapping only speaker identity.","Because the linguistic targets come from MMS, the method's benefit likely depends on how well MMS covers the target language; a testable prediction is that CER and UTMOS degrade for low-resource languages under-represented in MMS's 1,400-language pretraining.","The binary pitch and energy targets discard the magnitude of prosodic movement; using continuous, speaker-normalized targets might preserve the disentanglement while capturing more expressive contours.","The slight drop in speaker similarity relative to CrossSpeech suggests a controllable trade-off: weighting LD versus SD outputs could let a user tune between accent fidelity and voice identity."],"forward_implications":["Cross-lingual synthesis on monolingual corpora can approach intra-lingual quality: the paper reports its cross-lingual MOS (4.06) exceeding all baselines and its CER (13.35) below them.","Adding a new language does not require a pre-computed aligner, because the online duration aligner produces ground-truth durations inside training.","The SSL-based linguistic adaptor is the single most important module: removing it drops cross-lingual UTMOS from 3.791 to 3.443 in the paper's ablation.","The system also improves intra-lingual synthesis, so the decoupled generators are not just a cross-lingual fix but a general TTS design.","Compared with zero-shot systems, the paper's model yields higher naturalness (UTMOS 3.863 vs 3.450 for XTTS-v2) and speaker similarity (SECS 0.767 vs 0.763) when both are evaluated on English, Chinese, and Japanese."],"supporting_citations":[{"why":"Supplies the FastPitch backbone architecture and serves as the primary baseline.","marker":"[7]"},{"why":"Is the previous CrossSpeech split-generator system that this work extends and directly compares against.","marker":"[38]"},{"why":"Provides the online duration aligner that removes the need for per-language external aligners.","marker":"[41]"},{"why":"Is the frozen MMS model whose last hidden layer supplies the linguistic adaptor's target features.","marker":"[53]"},{"why":"Defines the wav2vec2.0 framework that MMS is built on, grounding the choice of SSL features.","marker":"[54]"},{"why":"Defines the information perturbation used to strip speaker cues from waveforms.","marker":"[50]"},{"why":"Inspires MDSLN with the MixStyle technique of mixing feature statistics across domains.","marker":"[40]"},{"why":"Provides DSLN, the speaker-conditioned layer normalization used in the speaker-dependent generator.","marker":"[45]"},{"why":"Is a zero-shot cross-lingual baseline used in the comparison reported in Table V.","marker":"[74]"},{"why":"Is a zero-shot cross-lingual baseline used in the comparison reported in Table V.","marker":"[75]"}],"fun_headline_variants":["Split language and speaker generators for better cross-lingual speech","Decoupled generators lift cross-lingual speech synthesis quality","Two-module design disentangles language and speaker in speech synthesis","CrossSpeech++: Separate language and speaker for natural multilingual voice","Language–speaker disentanglement achieves big gains in cross-lingual synthesis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the last hidden layer of the MMS model, applied to waveforms after formant shifting, pitch randomization, and frequency shaping, contains the linguistic content with speaker identity largely removed; if those features still carry residual speaker or language artifacts, the language-dependent generator will encode entangled information and the disentanglement claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Split language and speaker generators for better cross-lingual speech","Decoupled generators lift cross-lingual speech synthesis quality","Two-module design disentangles language and speaker in speech synthesis","CrossSpeech++: Separate language and speaker for natural multilingual voice","Language–speaker disentanglement achieves big gains in cross-lingual synthesis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1491,"prompt_tokens":974,"completion_tokens":517,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":429}},"tokens_in":590,"tokens_out":517,"duration_ms":5056,"temperature":1.0,"reasoning_tokens":429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:36:13.016194+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to train a speaker classifier on the language-dependent features produced by the LDG for a held-out set of speakers and languages: if it can predict speaker identity above chance, the LD branch has not actually removed speaker information. A complementary check is to swap only the language embedding while holding the speaker fixed and measure speaker-embedding cosine similarity of the output; a large drop in similarity would indicate speaker leakage into the language branch.","supporting_citations":[{"cited_title":"Fastpitch: Parallel text-to-speech with pitch prediction,","cited_arxiv_id":null,"evidence_quote":"Supplies the FastPitch backbone architecture and serves as the primary baseline."},{"cited_title":"Crossspeech: Speaker-independent acoustic representation for cross- lingual speech synthesis,","cited_arxiv_id":null,"evidence_quote":"Is the previous CrossSpeech split-generator system that this work extends and directly compares against."},{"cited_title":"One TTS alignment to rule them all,","cited_arxiv_id":null,"evidence_quote":"Provides the online duration aligner that removes the need for per-language external aligners."},{"cited_title":"Scaling speech technology to 1,000+ languages,","cited_arxiv_id":null,"evidence_quote":"Is the frozen MMS model whose last hidden layer supplies the linguistic adaptor's target features."},{"cited_title":"Wav2vec 2.0: A framework for self-supervised learning of speech representations,","cited_arxiv_id":null,"evidence_quote":"Defines the wav2vec2.0 framework that MMS is built on, grounding the choice of SSL features."},{"cited_title":"Neural analysis and synthesis: Reconstructing speech from self-supervised representations,","cited_arxiv_id":null,"evidence_quote":"Defines the information perturbation used to strip speaker cues from waveforms."},{"cited_title":"Domain generalization with mixstyle,","cited_arxiv_id":null,"evidence_quote":"Inspires MDSLN with the MixStyle technique of mixing feature statistics across domains."},{"cited_title":"PV AE-TTS: Adaptive text-to-speech via progressive style adaptation,","cited_arxiv_id":null,"evidence_quote":"Provides DSLN, the speaker-conditioned layer normalization used in the speaker-dependent generator."}],"review_version":1}