REVIEW 4 major objections 4 minor 7 references
Open-Source Full-Duplex Conversational Datasets for Natural and Interactive Speech Synthesis
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper offers open-source dual-track conversation datasets and shows they make a TTS model sound more natural.
desk verdict A useful dataset contribution whose fine-tuning validation is underreported to the point of being unverifiable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the dual-track recording and annotation scheme. Recording each participant on a separate mono track preserves true simultaneous speech (overlaps, barge-ins, backchannels) that mixed recordings lose; the annotations then mark overlapping intervals explicitly, letting a TTS model learn turn-taking dynamics rather than sequential turns only. The validation pipeline uses a pretrained TTS model with a cross-entropy loss for text-to-token generation and an optimal-transport conditional-flow-matching loss for mel spectrogram generation, fine-tuned separately on each language corpus.
What would settle it
Take the released corpora, fine-tune the same TTS model exactly as described, then evaluate on conversations from speakers withheld from fine-tuning and on transcripts never seen in training. If the F0, spectrum, ZCR, and A/B preference gains shrink to noise, the claim that these datasets improve conversational naturalness is disproved; if they persist, the central claim is supported.
Extended reading notes
Core claim
The central claim is that a modest amount (15 hours total) of carefully recorded, speaker-separated spontaneous conversation can push a conversational TTS system toward more natural, interactive-sounding speech. The datasets record pairs in separate rooms on dual mono tracks, so simultaneous speech is physically separated rather than mixed; transcripts mark speaker, gender, timestamps, overlapping intervals, and non-verbal vocal actions. After fine-tuning, the Chinese model reduced F0 Wasserstein distance by 7.08% and spectrum distance by 1.59%, while the English model reduced F0 distance by 3.67% and ZCR distance by 7.44%; listener preference (A/B) favored the fine-tuned model by 45.0% (Chi
Load-bearing premise
The load-bearing premise is that the objective and subjective evaluations used conversation segments the model had not already been fine-tuned on; the paper never discloses a train/test split, so if the test clips came from training conversations, the measured gains could be memorization rather than general improvement.
Editorial extensions
If this is right
- If these corpora are adopted, conversational TTS can be trained on true full-duplex phenomena instead of cleaned turn-taking data.
- The reported metric gains suggest that even 5–10 hours of such data can shift prosody, pitch, and zero-crossing distributions toward real speech.
- The datasets provide both audio tracks and timestamped transcripts, so downstream systems can predict when to barge in or yield the floor.
- Open release under CC BY 4.0 with supporting fine-tuning and evaluation code makes the claim reproducible and the corpora reusable.
Reading between the lines
- The paper's comparisons are against the same model before fine-tuning; a stronger test would also compare against a model fine-tuned on an equal amount of turn-based conversational data, isolating the full-duplex annotations as the cause.
- If evaluation used training conversations (the paper does not state a split), the improvements may partly reflect memorization; held-out and cross-speaker evaluation would reveal the generalization gain.
- The same corpora could be repurposed for overlap-aware voice activity detection, turn-taking prediction, or dialogue state modeling, not just TTS.
- The very small MOS differences suggest the practical benefit may vary by downstream use; the main value may be the annotated overlap structure rather than raw naturalness.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents two dual-track conversational speech datasets, one Chinese (~10 h) and one English (~5 h), collected by recording each interlocutor on a separate mono track in isolated rooms, with transcripts and annotations for speaker identity, timestamps, and overlapping speech. To demonstrate utility, the authors fine-tune CosyVoice-300M on each language subset (200 epochs, learning rate 1e-5) and report objective metrics (Table 2), A/B preference results (Figure 2), and MOS scores (Table 3), claiming consistent improvements in naturalness and conversational realism. The datasets are intended to fill a gap in full-duplex conversational TTS training resources.
Significance. Full-duplex conversational speech data with separate speaker tracks and overlap/backchannel annotations are genuinely scarce, and the collection protocol—familiar dyads, free topics, isolated rooms—is well motivated for ecological validity. If the datasets are released as promised and validated on held-out data, they would be a useful resource for conversational TTS and spoken-dialogue research. However, the current empirical validation is not yet sufficient to establish the central claim that fine-tuning on these data improves synthesis: the evaluation split is undisclosed, one objective metric directly contradicts the 'consistent improvements' assertion, and the subjective results lack statistical support. The contribution is conditional on addressing these issues.
major comments (4)
- [Section 4 (Technical Validation), Tables 2–3] The manuscript never discloses a train/evaluation split. It states that CosyVoice is fine-tuned on the Chinese and English datasets and then reports objective and subjective scores without indicating whether the synthesized test utterances come from conversations held out from fine-tuning. If the same conversation segments are used, the reported gains (e.g., Chinese F0 Wasserstein −7.08%, English ZCR −7.44%) could reflect memorization rather than generalization. Please specify a conversation-level split and report all metrics on held-out conversations; this is load-bearing for the abstract's claim of improved naturalness.
- [Section 4.1, Table 2] The text says 'The evaluation revealed consistent improvements across both Chinese and English subsets,' but the English Spectrum l2 row shows a degradation (34.5732 → 34.8681, −0.85%). This is a direct contradiction of the 'consistent improvements' claim. Please correct the wording and either explain the regression or provide a corrected metric. This is not merely cosmetic: it affects the central assertion that fine-tuning improves acoustic similarity.
- [Section 4.2, Table 3 and Figure 2] Subjective evaluation uses only 10 listeners per language, with no significance tests, confidence intervals, or error bars. The MOS differences (e.g., Chinese naturalness 4.37 vs. 4.40; English intelligibility 4.56 vs. 4.58) and A/B margins (Chinese 45.0% vs. 33.9%; English 46.4% vs. 40.7%) are small relative to expected listener variance. Without a statistical analysis, the claim of 'perceptible improvements' is not supported. Add appropriate significance testing or temper the conclusion to reflect the exploratory nature of the result.
- [Section 5, Data Availability Statement] The persistent link is given as '[DOI/URL]' and no actual repository, DOI, or code link is provided for the datasets or for the 'supporting code for fine-tuning and evaluation' promised in the abstract. Since the paper's primary deliverable is an open-source dataset, this placeholder prevents external verification of the central contribution. The final version must include a working link/DOI.
minor comments (4)
- [Section 2.2] Typographical issues: 'V AD' should be 'VAD', 'W A V' should be 'WAV', and 'V oice' should be 'Voice'. Please proofread the text for similar spacing artifacts.
- [Table 1] The English subset has 4149 utterance segments in 5 hours, while the Chinese subset has 2306 segments in 10 hours. This is surprising and should be clarified, e.g., by defining the utterance-segmentation and merging criteria used for each language.
- [Section 4.1] The phrase 'significant reduction' is used colloquially (e.g., 'the Chinese model exhibited a significant reduction in F0 distance'). Since no statistical significance testing is reported, please replace 'significant' with 'reported' or add proper tests.
- [Section 4.2] The description 'Each listener evaluated the same 80 pairs of conversation' is unclear: does each pair correspond to a distinct conversation segment or a repeated text prompt? Please clarify the stimulus construction and whether these pairs are from the held-out set.
Circularity Check
No circular derivation: the dataset-utility claim rests on external fine-tuning evaluation, not on equations that reduce to fitted inputs.
full rationale
The paper's central claim is an empirical benchmark result: fine-tuning CosyVoice-300M on the proposed full-duplex conversation corpora improves objective and subjective metrics relative to the pretrained baseline. No equation in the paper defines a prediction in terms of a fitted parameter or evaluation metric. Equations (1) and (2) are standard CosyVoice training losses (cross-entropy and optimal-transport conditional flow matching); they are not used to define the evaluation metrics, nor are the evaluation metrics algebraically derived from them. The objective metrics (spectrum l2, F0/energy/ZCR Wasserstein distance) and subjective A/B and MOS ratings are external comparisons between synthesized and real speech, and the baseline CosyVoice is an external system, not a self-citation. There is no imported uniqueness theorem, no ansatz smuggled via citation, and no renaming of a known result. The only substantive concern is that Section 4 does not report a train/evaluation split: if the evaluated clips were generated from the same conversation segments used for fine-tuning, the reported improvements could reflect memorization rather than generalization. However, the text does not state that evaluation used training segments, and this is a methodological-validity issue rather than a circular-derivation burden. The placeholder data link in Section 5 also affects reproducibility but is not circularity. The derivation chain is therefore self-contained relative to the external baseline, and no circular step is identifiable from the paper's own statements.
Assumptions & free parameters
free parameters (3)
- Learning rate for CosyVoice fine-tuning =
1e-5
- Training epochs for fine-tuning =
200
- Maximum utterance length for training =
30 characters (Chinese), 15 words (English)
assumptions (3)
- domain assumption Recruiting pairs who know each other yields naturalistic full-duplex speech.
- domain assumption The acoustic metrics (spectrum l2, F0/energy/ZCR Wasserstein) correlate with perceived naturalness.
- domain assumption The pre-trained CosyVoice-300M is a reasonable baseline for conversational TTS.
Cite this review
Pith. "Pith review of Open-Source Full-Duplex Conversational Datasets for Natural and Interactive Speech Synthesis." pith.science (2026). https://pith.science/paper/XYQCW6QC
@misc{pith2026250904093,
author = {Pith},
title = {Pith review of: Open-Source Full-Duplex Conversational Datasets for Natural and Interactive Speech Synthesis},
year = {2026},
howpublished = {\url{https://pith.science/paper/XYQCW6QC}},
note = {Machine review of arXiv:2509.04093}
}
read the original abstract
Full-duplex, spontaneous conversational data are essential for enhancing the naturalness and interactivity of synthesized speech in conversational TTS systems. We present two open-source dual-track conversational speech datasets, one in Chinese and one in English, designed to enhance the naturalness of synthesized speech by providing more realistic conversational data. The two datasets contain a total of 15 hours of natural, spontaneous conversations recorded in isolated rooms, which produces separate high-quality audio tracks for each speaker. The conversations cover diverse daily topics and domains, capturing realistic interaction patterns including frequent overlaps, backchannel responses, laughter, and other non-verbal vocalizations. We introduce the data collection procedure, transcription and annotation methods. We demonstrate the utility of these corpora by fine-tuning a baseline TTS model with the proposed datasets. The fine-tuned TTS model achieves higher subjective and objective evaluation metrics compared to the baseline, indicating improved naturalness and conversational realism in synthetic speech. All data, annotations, and supporting code for fine-tuning and evaluation are made available to facilitate further research in conversational speech synthesis.
Figures
Reference graph
Works this paper leans on
-
[1]
Lin, G.-T. et al. (2025). Full-duplex speech models—systems that can speak and listen concurrently—are reshaping the landscape of spoken human–computer interaction. arXiv preprint, arXiv:2507.23159
arXiv 2025
-
[2]
Arora, S. et al. (2025). Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics. arXiv preprint arXiv:2503.01174
arXiv 2025
-
[3]
Ma, Z. et al. (2024). Language model can listen while speaking. arXiv preprint,å arXiv:2408.02622
arXiv 2024
-
[4]
,The CANDOR corpus: Insights from a large multimodal dataset of naturalistic conversation
Andrew Reece et al. ,The CANDOR corpus: Insights from a large multimodal dataset of naturalistic conversation. Sci. Adv.9,eadf3197(2023)
work page 2023
-
[5]
J. J. Godfrey, E. C. Holliman and J. McDaniel, ”SWITCHBOARD: telephone speech corpus for research and de- velopment,” [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, San Francisco, CA, USA, 1992, pp. 517-520
work page 1992
-
[6]
Lee, D. et al. DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech. arXiv preprint, arXiv: 2207.01063 (2023)
work page Pith review arXiv 2023
-
[7]
X. Lyu, Y . Wang, T. Zhao, H. Wang, H. Liu and Z. Du, ”Build LLM-Based Zero-Shot Streaming TTS System with CosyV oice,” ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–2. 6 Data Intelligence
work page 2025
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.