{"id":"1e82ef7d-ea44-46c1-8f16-6f36c4f72350","arxiv_id":"2507.15272","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A2TTS adds a reference-audio cross-attention duration predictor to a Grad-TTS and UnitSpeech style diffusion TTS, improving speaker similarity scores in seven Indian languages.","lead":"A2TTS is a text-to-speech system that can synthesize speech in a new speaker's voice from a short reference audio sample, trained for seven Indian languages. It combines a diffusion speech generator, a speaker encoder, and a cross-attention duration predictor that borrows rhythm cues from the reference clip.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training computes the speaker embedding from the target utterance itself, whereas inference uses a separate reference; this train/inference mismatch means the reported Sim-O numbers do not demonstrate zero-shot speaker adaptation.","rationale":"The paper's claim to outperform existing models in zero-shot speaker adaptation rests on experiments where the speaker embedding used during training is extracted from the target utterance, not from a same-speaker reference as in inference. This is not a minor implementation detail: it changes the information available to the model. A speaker encoder trained on VoxCeleb2 is not guaranteed to discard utterance-level prosody and content; if any leaks through, the model can use it to regenerate the exact target prosody, making training easier and making the test-time reference condition out-of-distribution. This directly affects the validity of every reported Sim-O and intelligibility number. The reader's identified weak assumption, that the English-trained UnitSpeech encoder transfers to Indic languages, is also real, but it is a separate external-generalization risk. The train/inference mismatch is an internal flaw that can be settled by a matched-protocol retraining. I therefore recommend UNVERDICTED rather than CONDITIONAL: the current paper does not provide evidence for the central claim as stated, though a corrected experiment could still vindicate the architecture.","tokens_in":6987,"tokens_out":10103,"duration_ms":124138,"concrete_test":"Retrain A2TTS and the GradTTS baseline with a matched zero-shot protocol: for each training text, sample a different utterance from the same speaker, compute es from that utterance's mel using the frozen UnitSpeech encoder, use a second different utterance as the duration reference M, and train the model to synthesize the target text. Evaluate both models on the same held-out IndicSUPERB test set. If the Sim-O gains over the baseline shrink or vanish, the reported results were driven by target-utterance leakage; if they persist, the zero-shot claim is supported. Also report test results with es extracted from full-length versus 2-second references to check the feature-statistics mismatch.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim is zero-shot speaker adaptation, but the training procedure described in the Figure 1 caption and Section 2.2 is not the inference procedure. During training, the speaker embedding es is computed from the target speech—the very ground-truth utterance corresponding to the synthesized text—while a separate reference mel M is used only in the duration predictor. During inference, a single reference audio is used for both es and M. Thus the model is trained with a speaker embedding that can contain target-utterance prosody and content, and with feature statistics from a full target utterance, then evaluated with embeddings from a 2-second unrelated reference. If the UnitSpeech encoder is not perfectly utterance-invariant, the diffusion decoder and duration predictor can exploit this oracle signal during training. Consequently, the Sim-O gains in Table 2 and the CER numbers in Table 1 cannot be attributed to a genuine zero-shot capability; they may instead reflect a distribution shift at test time. Because the GradTTS baseline is presumably trained with the same protocol, the relative comparison is also unreliable: the baseline may be equally mismatched, and no experiment isolates the claimed benefit of the cross-attention duration predictor under matched training and inference conditions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes A2TTS, a speaker-conditioned diffusion TTS built on Grad-TTS and UnitSpeech, whose main novelty is a cross-attention duration predictor that conditions phoneme durations on a reference mel spectrogram, plus classifier-free guidance at inference. The authors train language-specific models on IndicSUPERB (with IndicTTS added for Malayalam and Tamil) and evaluate speaker similarity (Sim-O) and intelligibility (ASR CER/WER) across seven Indian languages. The central claim is improved zero-shot speaker adaptation over a GradTTS baseline conditioned on speaker embeddings.","tokens_in":7167,"tokens_out":6897,"duration_ms":59586,"significance":"If the evaluation were sound, the contribution would be a modest but useful extension for low-resource Indic TTS: the cross-attention duration conditioning is simple and could improve prosody for unseen speakers. The paper has strengths: it evaluates across multiple languages and multiple external datasets (IndicSUPERB, FLEURS, CommonVoice, IndicTTS, Kathbath), and it reports CER on both ground-truth and synthesized audio, which is informative for intelligibility. However, the current evidence does not establish the central claim because of a train/inference speaker-embedding mismatch, a confounded baseline, lack of statistical significance, and no human listening test. The paper also does not release code or audio samples, which limits reproducibility.","major_comments":[{"comment":"During training, the speaker embedding es is computed from the target speech that contains the target transcript, while a separate reference mel M is used only for cross-attention; during inference, es and M both come from a single 2-second reference (Figure 1 caption, Section 2.2). This gives the training-time model oracle access to the exact utterance being synthesized, and if the UnitSpeech encoder is not perfectly utterance-invariant, the Sim-O gains in Table 2 and the CER numbers in Table 1 can reflect target-utterance leakage rather than zero-shot speaker adaptation. Because the GradTTS baseline is presumably trained with the same protocol, the relative comparison is also unreliable. Please retrain and evaluate under a matched protocol (e.g., always derive es from a distinct same-speaker reference at both training and inference) or provide evidence that es is utterance-invariant, such as same-speaker different-utterance and different-speaker embedding similarity distributions.","section":"Section 2.2, Figure 1 caption, Table 2"},{"comment":"The baseline 'Gradtts conditioned on speaker embeddings' differs from A2TTS in multiple respects: double the number of channels, 1,500 plus 1,000 training epochs, classifier-free guidance at inference, and the cross-attention duration predictor. The improvements in Table 2 therefore cannot be attributed to the proposed module. Please add ablations that hold all other components fixed and vary only the duration predictor, and also compare against the UnitSpeech system on which the paper builds. Report means and variances over multiple seeds or test splits; the gap for Gujarati (0.7221 vs 0.7250, about 0.003) is likely within noise.","section":"Section 3.2 and Table 2"},{"comment":"The table reports CER for A2TTS outputs but includes no baseline TTS CER, so the claim that the proposed approach improves intelligibility over existing models is not demonstrated. Additionally, the caption states that Malayalam and Tamil training includes IndicTTS, and the IndicTTS rows for those languages are evaluated on the IndicTTS test set; this is circular for the cross-dataset generalization claim. Please present matched train/test conditions with baseline CER under identical conditions, or remove the IndicTTS rows for Malayalam and Tamil from the cross-dataset comparison.","section":"Table 1 and its caption"},{"comment":"The Sim-O metric is computed as cosine similarity of speaker embeddings, but the paper does not specify which encoder is used. If it is the same UnitSpeech encoder used for conditioning, the metric is circular because the model is trained to make outputs close to that embedding. Please use an external speaker verification model, report human similarity MOS, and include a listening test.","section":"Section 4.1"}],"minor_comments":[{"comment":"The phrase 'allowing the system to generate speech more near speech for unknown speakers' is ungrammatical; suggest 'generating speech closer to the target speaker'.","section":"Abstract"},{"comment":"The notation M is used both for the reference mel spectrogram and as keys/values; clarify how the fixed-length segment is extracted and projected, since mel spectrograms vary in length.","section":"Section 2.2, Eq. (2)"},{"comment":"The caption 'Here the gradtts conditioned on speaker embeddings is presented as baseline and test set average is the simo over the test dataset of indicsuperb' should be rewritten; also use consistent capitalization for GradTTS and IndicSUPERB.","section":"Table 2 caption"},{"comment":"The phrase 'ASR error is the CER when decoding the original reference transcriptions (i.e., ground-truth synthesis CER)' is confusing; it should read 'CER on ground-truth audio'.","section":"Table 1 caption"},{"comment":"The statement that the cross-attention mechanism 'eliminates the need for explicitly training a separate duration model, as required in systems like Voicebox and F5-TTS' is not supported by the cited references; please either provide evidence or rephrase.","section":"Section 1"},{"comment":"The Vaswani et al. entry lacks year and venue, and several other entries have inconsistent formatting (e.g., Kunchukuttan et al., 2022 appears in two different forms).","section":"References"},{"comment":"The sentence 'we employ the speaker encoder from UnitSpeech, which is trained on VoxCeleb2 (Chung et al., 2018) a dataset comprising 6,112 speakers' has a punctuation error; also note that the encoder appears not to be fine-tuned on Indic data, which is relevant to the zero-shot claim.","section":"Section 3.1"},{"comment":"The abstract and Section 3.1 mention 12 languages in IndicSUPERB, but experiments cover only seven; please clarify how the seven were selected.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The train/inference mismatch described in the paper's own Figure 1 caption is the key obstacle: it is not an artifact of the review process but is stated explicitly in the manuscript. The authors should be asked to rerun the experiments under a matched protocol and to add ablations before the claims can be accepted. The paper also lacks comparisons with strong zero-shot baselines (YourTTS, UnitSpeech, Voicebox) and does not provide code or audio samples."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Three things up front. The only genuinely new component is the cross-attention duration predictor in Section 2.2, a small extension of UnitSpeech. The paper's real weakness is not the idea but the evaluation: during training the speaker embedding is computed from the target utterance, while inference uses a separate reference, so the reported Sim-O numbers come from a configuration the model was never trained in. And the baseline in Table 2 differs in three ways beyond the module—double channels, more epochs, and CFG—so the improvement is not a clean ablation of the cross-attention.\n\nWhat the paper does well: it targets a practical gap (low-resource Indic TTS), builds on solid components, and is clearly written. The authors are honest about limitations: training cost, restricted speaker diversity, out-of-domain adaptation. The cross-attention idea is plausible: conditioning duration on a reference mel to capture speaker prosody is a reasonable thing to try.\n\nThe soft spots compound. No error bars or significance tests, no listening test, no comparison against UnitSpeech itself. The Malayalam and Tamil rows include IndicTTS in training and evaluate on its test set, which the caption discloses but still inflates those numbers. The UnitSpeech speaker encoder, trained on English VoxCeleb2, is used as-is with no check that its embeddings separate Indic speakers—this is load-bearing because both the decoder and the duration predictor condition on it. The stress-test note about the train/inference mismatch is accurate. That mismatch does not necessarily invalidate the zero-shot test numbers, but it means the model is evaluated in a configuration it was never trained for, and the relative gain over the baseline is hard to interpret.\n\nMy overall take: this is a workshop-grade paper with a sensible idea and an unsupported headline claim. The core contribution—cross-attention duration conditioning—is worth testing, but the paper needs a matched baseline (UnitSpeech with the same training protocol), matched train/inference, uncertainty estimates, and at least one listening test. I would not cite this as evidence of SOTA zero-shot TTS, but I would read a revised version.\n\nWho is this for? Someone building TTS for Indian languages might find the recipe useful, and a referee's concrete guidance could sharpen the evaluation substantially. It deserves a serious referee, not a desk reject, but the verdict should be major revision.\n\nRecommendation: engage if the authors redo the evaluation; send to peer review with that expectation.","headline":"Plausible UnitSpeech variant whose zero-shot claim is undercut by a train/inference speaker-embedding mismatch and a non-ablative baseline.","tokens_in":7746,"tokens_out":3959,"would_cite":false,"duration_ms":44855,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a cross-attention duration predictor conditioned on a reference speaker's mel spectrogram, added to a Grad-TTS diffusion backbone with speaker embeddings and classifier-free guidance, improves zero-shot speaker…","keywords":["text-to-speech","zero-shot speaker adaptation","diffusion model","cross-attention duration prediction","low-resource Indian languages","classifier-free guidance","speaker similarity","IndicSUPERB"],"falsifier":"Ablate the cross-attention duration predictor by replacing the reference mel with one from a different speaker while holding the target speaker embedding fixed; if Sim-O and CER do not degrade, the central claim that duration prediction is speaker-adaptive is not supported.","tokens_in":6745,"feed_emoji":"🎙️","tokens_out":6001,"duration_ms":53422,"temperature":0.7,"pith_summary":"The paper sets out to show that a diffusion-based text-to-speech system can synthesize speech for unseen speakers in seven low-resource Indian languages with higher speaker similarity and intelligibility than the speaker-embedded GradTTS baseline it extends. Its central modification is a cross-attention duration predictor that takes the text embedding as queries and a short reference mel spectrogram from the target speaker as keys and values, so phoneme durations become speaker-dependent. The paper also applies classifier-free guidance at inference, using the dataset-wide mean mel spectrogram as the unconditional condition. If correct, the approach would make it possible to build a personalized TTS for a new Indian-language voice from a couple of seconds of untranscribed audio, without fine-tuning.","feed_headline":"Cross-attention durations lift zero-shot TTS across 7 Indian languages","feed_subtitle":"Reference-audio duration attention lifts speaker similarity and intelligibility over a speaker-embedded GradTTS baseline.","key_machinery":"The load-bearing mechanism is the cross-attention duration predictor, written $A = \\mathrm{Attention}(E_t, M, M)$ with text-embedding queries $E_t$ and a fixed-length reference mel spectrogram $M$ as both keys and values, followed by a duration network $D = f_{dur}(A)$. The reference $M$ is a two-second segment from the target speaker deliberately chosen to be unrelated to the input text, which prevents the model from memorizing text-dependent durations instead of learning speaker-specific timing. The speaker encoder from UnitSpeech provides the embedding $e_s$ used to condition the diffusion decoder, and classifier-free guidance with the dataset-wide mean mel as the unconditional condition is applied during inference.","core_discovery":"A2TTS claims that conditioning duration prediction on reference speech—not just conditioning the diffusion decoder on speaker embeddings—is what carries zero-shot speaker adaptation. On a test set of 20 unseen speakers per language from IndicSUPERB, the proposed model raises Sim-O speaker-similarity scores across Hindi, Marathi, Gujarati, Punjabi, Bengali, Tamil, and Malayalam compared with a GradTTS model conditioned on speaker embeddings, and it reports lower character error rates on synthesized speech over several evaluation corpora. The paper interprets these gains as evidence that the cross-attention duration predictor aligns linguistic features with speaker-specific timing and prosody, while classifier-free guidance sharpens pronunciation and speaker consistency for unseen voices.","pith_inferences":["The paper does not test whether the English-trained speaker encoder transfers to Indian languages; measuring speaker-verification accuracy of that encoder on IndicSUPERB speakers would directly test the foundation of the conditioning.","Because the duration predictor is conditioned on a reference mel from the same speaker, an untested but natural stress test is whether varying the reference's speaking rate or emotion changes predicted durations accordingly, which would confirm it learns prosody rather than just identity.","A single multilingual model with language tokens, instead of seven language-specific models, would be the natural next step and might generalize better to low-resource languages with even less data."],"forward_implications":["A user could clone a new voice in any of the seven supported languages from a two-second reference clip, without any per-speaker fine-tuning.","Speaker-similarity scores (Sim-O) on held-out speakers improve for all seven languages relative to the speaker-embedded GradTTS baseline.","The cross-attention duration predictor is trained end-to-end, so the system does not need a separately trained duration model as in Voicebox or F5-TTS.","Classifier-free guidance improves pronunciation and speaker consistency for unseen speakers while leaving the training procedure unchanged.","Intelligibility, measured as ASR character error rate on synthesized speech, is at least competitive with the baseline across several evaluation corpora, including FLEURS, CommonVoice, IndicTTS, and Kathbath."],"supporting_citations":[{"why":"Grad-TTS diffusion backbone that A2TTS extends; supplies the text encoder, duration predictor, and DDPM decoder.","marker":"Popov et al., 2021"},{"why":"UnitSpeech; supplies the speaker encoder, decoder conditioning, and classifier-free-guidance implementation A2TTS adopts.","marker":"Kim et al., 2023"},{"why":"VoxCeleb2; the English dataset the frozen speaker encoder was trained on, so the transfer to Indic voices is assumed.","marker":"Chung et al., 2018"},{"why":"IndicSUPERB; the dataset providing training and test speakers across the seven Indian languages, and the held-out test speakers for Sim-O evaluation.","marker":"Javed et al., 2022"},{"why":"HiFi-GAN; the vocoder used to convert predicted mels to audio for all models in the comparison.","marker":"Kong et al., 2020"},{"why":"Classifier-free diffusion guidance; the inference-time technique that strengthens the text and speaker conditions.","marker":"Ho and Salimans, 2022"},{"why":"IndicTTS; additional training data used for Malayalam and Tamil, improving robustness.","marker":"Srivastava et al., 2020"}],"fun_headline_variants":["Reference-audio durations boost zero-shot TTS in 7 Indian languages","Speaker-aware durations improve zero-shot TTS across Indian languages","Zero-shot TTS for 7 Indian languages gets duration boosts from reference audio","Diffusion TTS conditions durations on reference speech for 7 Indian languages","Cross-attention duration conditioning lifts zero-shot TTS in 7 Indian languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole speaker-conditioning pipeline rests on speaker embeddings from an encoder trained only on English VoxCeleb2; if those embeddings do not carry over to Indian languages, both the diffusion conditioning and the cross-attention duration predictor lose their speaker signal.","fun_headline_variants_meta":{"raw":{"variants":["Reference-audio durations boost zero-shot TTS in 7 Indian languages","Speaker-aware durations improve zero-shot TTS across Indian languages","Zero-shot TTS for 7 Indian languages gets duration boosts from reference audio","Diffusion TTS conditions durations on reference speech for 7 Indian languages","Cross-attention duration conditioning lifts zero-shot TTS in 7 Indian languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000843,"raw_usage":{"total_tokens":3618,"prompt_tokens":841,"completion_tokens":2777,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":2682}},"tokens_in":457,"tokens_out":2777,"duration_ms":19746,"temperature":1.0,"reasoning_tokens":2682,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:35:39.218460+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ablate the cross-attention duration predictor by replacing the reference mel with one from a different speaker while holding the target speaker embedding fixed; if Sim-O and CER do not degrade, the central claim that duration prediction is speaker-adaptive is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"IndicTTS; additional training data used for Malayalam and Tamil, improving robustness."}],"review_version":1}