{"id":"01f14919-ba37-44bd-9b22-6baf195c4031","arxiv_id":"2504.14482","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A three-agent script-writer, synthesizer, and critic pipeline generates a bilingual multi-party speech dataset whose quality matches manually assembled datasets in TTS training benchmarks.","lead":"This paper presents DialogueAgents, a three-agent system that writes scripts, turns them into synthetic speech, and critiques the audio to generate a bilingual multi-party dialogue dataset called MultiTalk. Readers may care because it points to a cheaper, automated way to build conversational speech data for training models that talk naturally.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset evaluation is circular: MultiTalk is both training and test distribution, so 'comparable MOS' does not establish utility.","rationale":"The Reader's weakest_assumption—using GPT-4o as both script writer and script evaluator—is a legitimate concern for the script-quality tables, but it is not the most load-bearing issue for the paper's central contribution. The headline evidence for dataset utility is Table VI, and that evidence is confounded: MultiTalk-trained models are evaluated on MultiTalk, whose audio was generated by the same CosyVoice pipeline used to create the training data. This makes the 'comparable MOS' claim uninterpretable as evidence that MultiTalk is useful for training TTS on natural speech. The proposed cross-corpus test would settle the point: if a MultiTalk-trained model fails on natural DailyTalk test audio, then MultiTalk currently only demonstrates self-mimicry. The paper does release code and dataset, the framework description is clear, and the comparisons within a single dataset are internally consistent; the issue is the evaluation protocol, not the construction. Therefore the verdict should remain conditional, but for a different reason than the Reader emphasized: acceptance requires redoing the dataset evaluation with cross-corpus or natural-speech generalization evidence. The malformed '/uni...' strings in the source appear to be rendering artifacts and are not load-bearing.","tokens_in":9621,"tokens_out":7484,"duration_ms":71538,"concrete_test":"Retrain the same backbones (Tacotron2/FastSpeech2/DailyTalk-Model) on MultiTalk, then evaluate on the natural human-speech test split of DailyTalk; compare with a model trained on DailyTalk evaluated on the same DailyTalk test split, matching optimizer, steps, and effective epochs. If the MultiTalk-trained model's MOS on natural DailyTalk audio is more than 0.2 lower than the DailyTalk-trained model's, the V-C claim is an artifact of train/test distribution overlap. As a secondary check, evaluate a MultiTalk-trained model on held-out natural recordings from the same 30 characters instead of CosyVoice output.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that MultiTalk is a usable training resource, evidenced by 'MultiTalk achieves a similar MOS value despite having less than a quarter of the data volume' (Sec. V-C)—rests entirely on Table VI. In that experiment, each model is trained and then evaluated on held-out audio from the same dataset. In the 'Ours' column, both training and test audio are CosyVoice-generated synthetic speech from the same pipeline that built MultiTalk; the DailyTalk column uses natural human recordings. A model trained on MultiTalk can thus score well by reproducing CosyVoice's voice pool and artifacts, not by learning general dialogue-to-speech ability. The claimed 'similar MOS' therefore compares incompatible targets: synthetic-to-synthetic versus natural-to-natural. This is more load-bearing than the script-evaluator bias the Reader identified: even if GPT-4o self-refinement were unbiased, Table VI would still not establish that MultiTalk substitutes for human-recorded dialogue data. The comparison is also under-specified: Tacotron2/FastSpeech2 are trained for 900K steps while the DailyTalk-Model is trained 'on 850 dialogues ... until convergence,' and no test-set sizes or significance tests are reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DialogueAgents, a hybrid agent-based framework for generating multi-party, multi-turn speech dialogue data. The framework comprises three agents: a Script Writer (GPT-4o) that generates and refines dialogue scripts, a Speech Synthesizer (CosyVoice) that converts scripts to speech, and a Dialogue Critic (Qwen2-Audio) that reviews the synthesized speech and provides textual feedback for iterative refinement. The authors use this framework to construct MultiTalk, a bilingual (Chinese and English) dataset of 4,437 utterances across 30 characters, and introduce two dialogue-level metrics, Emotional MOS (EMOS) and Turn-taking MOS (TMOS). Experiments evaluate the framework through human MOS, automated UTMOS/WER/CER, LLM-based script naturalness and emotiveness scores, and downstream training of Tacotron2, FastSpeech2, and DailyTalk-Model on the generated dataset.","tokens_in":9868,"tokens_out":3718,"duration_ms":34845,"significance":"If the claims are validated, the contribution is valuable: an automated pipeline that generates multi-party dialogue speech data with emotional and paralinguistic richness could substantially reduce dataset construction cost and increase diversity relative to manually recorded corpora. The framework is simple, modular, and the authors release the dataset and code, which would facilitate reproducibility and follow-up work. The introduction of EMOS and TMOS as dialogue-level evaluation metrics is also a useful addition. The paper's strengths include a concrete end-to-end system, a public dataset release, and a downstream TTS training evaluation that, while currently flawed in design, demonstrates a genuine attempt to assess practical utility.","major_comments":[{"comment":"The dataset evaluation is circular and does not support the claim that MultiTalk substitutes for human-recorded dialogue data. Models trained on MultiTalk are evaluated on held-out MultiTalk audio, which is synthetic speech generated by the same CosyVoice pipeline and voice pool used to create the training set; models trained on DailyTalk are evaluated on held-out DailyTalk natural human speech. A high MOS in the 'Ours' column can be achieved by reproducing CosyVoice's voice characteristics and artifacts, rather than by learning general dialogue-to-speech ability. The comparison is therefore between in-domain synthetic-to-synthetic and in-domain natural-to-natural matches, which are not commensurate. To substantiate the claim, the authors should include a cross-dataset evaluation, such as training on MultiTalk and testing on natural speech (e.g., DailyTalk) or reporting human listening tests on natural reference audio, or at minimum clarifying what the 'Ground Truth' row means for a fully synthetic dataset.","section":"§V-C, Table VI"},{"comment":"The script naturalness and emotiveness scores in Table III are produced by GPT-4o, which is also the model that generates and refines the scripts. This self-evaluation setup risks systematic bias: GPT-4o may rate its own revised scripts higher regardless of genuine quality, and the reported improvements from the critic loop could reflect the evaluator's preference for its own output style rather than actual gains. This is load-bearing because Table III is the principal evidence that the iterative critic loop improves script quality. The authors should add human evaluation of the scripts or use an independent LLM as the evaluator, and report agreement between the automatic evaluator and human judgments.","section":"§IV-A, Script evaluation metrics"},{"comment":"The choice of T=2 as the number of refinement iterations is made after inspecting the ablation results on the same 30 dialogues and is then presented as an 'optimal number of iterations' finding. This is a fitted hyperparameter, not a validated principle. Moreover, the differences between the 1-loop, 2-loop, and 3-loop conditions overlap within the reported confidence intervals (e.g., MOS 3.71±0.063, 3.75±0.071, 3.78±0.075 in Table II), and no significance tests are reported. The claim of an optimal iteration count requires either statistical significance testing or validation on held-out dialogues; otherwise the selection of T=2 should be described as a design choice rather than an empirical discovery.","section":"§IV-B, Iteration count selection"}],"minor_comments":[{"comment":"The human evaluation details are incomplete: the paper states that 20 bilingual volunteers rated dialogue samples but does not specify the number of rated samples per condition, the rating instructions, the presentation order/randomization, or inter-rater agreement (e.g., Krippendorff's alpha). These details are needed to assess the reliability of the MOS, EMOS, and TMOS scores.","section":"§IV-A, Human evaluation"},{"comment":"The automated metrics (UTMOS, WER, CER) are reported as point estimates without confidence intervals or significance tests, and Table VI does not report test-set sizes. This makes it difficult to determine whether the observed differences are meaningful; please add uncertainty measures and test-set descriptions.","section":"Tables II and VI"},{"comment":"Figure 2 is described as an 'Ablation of critic agent' showing that refined scripts outperform originals in 75.8% of paired human judgments, but the figure's axes, the number of paired samples, and any statistical test are not described. Please clarify what the figure displays and how the 75.8% figure was computed.","section":"Figure 2"},{"comment":"The notation for the character pool is inconsistent and contains formatting artifacts (e.g., 'P oola', 'P oolp' with odd spacing). Please standardize the notation, for example using P_a, P_p, and P_c.","section":"§III-A"},{"comment":"The CosyVoice footnote contains a line break and the URL appears to be broken across lines, and the WenetSpeech4TTS reference is cited but the character pool description says the data come from both WenetSpeech4TTS and Common Voice; please verify the citations and clean the formatting.","section":"Footnotes and references"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and the core idea is interesting, but the central dataset-utility claim rests on a circular evaluation that needs to be reworked. The self-evaluation of scripts and the post-hoc selection of T=2 are additional load-bearing weaknesses that should be addressed with additional experiments rather than just textual revisions. I would encourage the editor to send the paper back for major revision rather than reject, because the framework and dataset are potentially useful and the issues are fixable with cross-dataset evaluation, human script evaluation, and significance testing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The pipeline—script writer (GPT-4o), synthesizer (CosyVoice2), critic (Qwen2-Audio) in an iterative loop—is a sensible reuse of agent collaboration for TTS dataset construction. MultiTalk is a new bilingual, multi-party, multi-turn dataset, and the code and data are public. That is a usable contribution.\n\nThe audit is right that Table VI is circular. Models are trained and evaluated on the same distribution: for MultiTalk, both sides are CosyVoice-generated synthetic speech; for DailyTalk, both sides are human recordings. A model can get a good MOS on MultiTalk by mimicking CosyVoice's voice pool and artifacts. So the “comparable MOS despite less data” claim does not demonstrate substitutability for human-recorded dialogue data. The DailyModel rows are a better comparison, but still the test distributions differ in difficulty. This is the load-bearing weakness, not the script-evaluator issue.\n\nThe second weakness is that script naturalness and emotiveness are judged by GPT-4o, the same model that wrote the scripts. That invites self-serving bias, and the fact that self-refinement alone improves scores under that judge suggests the gains may be partly inflated. Also, no significance tests are provided, most confidence intervals overlap, and the optimal iteration count T=2 is selected after seeing the ablation. These are secondary but real.\n\nWhat the paper does well: the critic-driven revision with paralinguistic tokens is a clean idea, the example shows genuine improvement, and the release of the dataset and pipeline is a useful resource for the community. The authors are straightforward about their setup.\n\nSo: worth serious peer review, not desk reject. But the revision needs to address the circular evaluation—ideally by adding a cross-dataset experiment where a model trained on MultiTalk is evaluated on natural dialogue audio, or at least by stating and contextualizing the limitation. A different LLM for script evaluation, or human ratings, would also strengthen the claims. This is a methods-and-resource paper for speech synthesis people; I'd bring it to a reading group if one of us works in TTS, otherwise it's a good cautionary example for evaluation design.","headline":"Useful pipeline and a new dataset, but the key utility claim rests on a circular evaluation; worth serious review with a required fix.","tokens_in":10353,"tokens_out":4674,"would_cite":false,"duration_ms":41538,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Three cooperating AI agents can generate multi-party dialogue speech good enough to train TTS models.","keywords":["speech synthesis","multi-party dialogue","multi-agent framework","dialogue dataset","paralinguistic tokens","iterative refinement","text-to-speech","MultiTalk"],"falsifier":"Have a panel of human raters score blinded dialogue scripts from the writer-only condition versus the two-loop critic condition on naturalness and emotiveness; if human ratings show no significant advantage for the critic-refined scripts, the claimed script-quality improvement collapses. Alternatively, train a TTS model on a random matched-size subset of DailyTalk and compare its MOS to a model trained on MultiTalk; if the subset-trained model matches or exceeds MultiTalk-trained MOS, the paper's central data-quality claim is falsified.","tokens_in":9431,"feed_emoji":"🎙️","tokens_out":2595,"duration_ms":24306,"temperature":0.7,"pith_summary":"This paper tries to establish that high-quality, multi-party, multi-turn spoken dialogue data can be produced automatically by a loop of three specialized agents—a script writer, a speech synthesizer, and a dialogue critic—rather than through costly human transcription and studio recording. If true, an automated pipeline can substitute for human-annotated dialogue data in training speech synthesis models, making customized, scalable dialogue data affordable. The authors use the framework to build MultiTalk, a bilingual English and Chinese dataset with 30 diverse characters, and report that models trained on it reach a MOS comparable to models trained on the human-recorded DailyTalk dataset, despite using less than a quarter of the data volume. The paper also argues that two iterations of critic feedback give the best quality, with a third iteration starting to degrade the output.","feed_headline":"Three-agent loop writes dialogue speech data that rivals hand-made sets","feed_subtitle":"MultiTalk matches DailyTalk-quality MOS with under a quarter of the data, pointing to cheaper TTS training data.","key_machinery":"The load-bearing mechanism is the critic-in-the-loop refinement cycle: a Script Writer agent (GPT-4o) generates a multi-party script from a pool of 30 hand-crafted character profiles, a Speech Synthesizer agent (CosyVoice2) converts it into audio, and a Dialogue Critic agent (Qwen2-Audio) listens and returns textual feedback on naturalness, clarity, and emotiveness. The writer then revises the script—inserting paralinguistic tokens like '[breath]' and '<strong>' and emotional labels like '[Agreeable]'—and the loop repeats. The character pool, with predefined social relationships, is what makes multi-party, multi-turn dialogues with consistent voices possible.","core_discovery":"The central claim is that an iterative three-agent loop—where a large language model writes dialogue scripts, a zero-shot speech synthesizer renders them with paralinguistic markers, and an audio-language model critic reviews the synthesized speech and feeds textual advice back to the writer—produces dialogue speech that is natural, emotionally expressive, and usable as training data. The authors demonstrate this by generating MultiTalk and evaluating it against established datasets. Their headline result is that a TTS model trained on MultiTalk achieves a mean opinion score statistically similar to one trained on DailyTalk, even though MultiTalk contains less than a quarter of the dialogues, supporting the idea that automatically synthesized dialogue data can rival human-recorded data in utility.","pith_inferences":["Because the same model (GPT-4o) both writes the scripts and evaluates their naturalness and emotiveness, the reported script-quality improvements from critic feedback could partly reflect self-serving bias; an independent evaluator or blinded human raters would give a cleaner test.","The critic's textual feedback could itself serve as a rich annotation signal—aligned with specific audio segments—potentially useful for training models on paralinguistic control beyond what MultiTalk's audio alone provides.","The iterative refine-and-synthesize loop is a general recipe that could extend to other domains, such as audiobook narration with multiple characters or interactive game dialogue, wherever a critic can listen and give actionable feedback.","A direct comparison against a human-recorded dataset of matched size, rather than the full DailyTalk, would isolate whether MultiTalk's quality claim comes from the data itself or from the TTS model's capacity to overfit a small, homogeneous set."],"forward_implications":["Automatically generated dialogue speech can serve as a viable substitute for human-annotated dialogue data in training TTS models, lowering the cost and effort of dataset construction.","Two iterations of critic feedback appear to be the sweet spot; adding a third iteration degrades quality due to over-optimization, suggesting an optimal stopping criterion for such loops.","The framework is model-agnostic per agent, so swapping in better script writers, synthesizers, or critics should yield further quality gains without redesigning the loop.","MultiTalk's bilingual, multi-party structure covers conversational scenarios—several speakers, interruptions, emotional shifts—that existing single-speaker or two-speaker TTS datasets lack.","The reported MOS parity with DailyTalk at one-quarter the data implies that data quality, not just quantity, drives conversational TTS performance."],"supporting_citations":[{"why":"DailyTalk is the primary dialogue-dataset baseline; its MOS and training setup anchor the claim that MultiTalk matches it with less than a quarter of the data.","marker":"[4]"},{"why":"VCTK supplies both a single-speaker TTS baseline and part of the character-pool audio used to craft character profiles.","marker":"[2]"},{"why":"WenetSpeech4TTS provides additional open-source audio for the character pool and represents the large-scale Mandarin TTS corpus the framework draws from.","marker":"[3]"},{"why":"The Fisher corpus serves as a prior multi-party conversational speech dataset with manual transcription, highlighting the cost problem MultiTalk aims to solve.","marker":"[5]"},{"why":"KE-Omni is the earlier sequential-agent synthetic dialogue dataset that the paper contrasts with its own iterative critic loop.","marker":"[6]"},{"why":"Common Voice contributes audio to the character pool, supporting the diversity of voices in MultiTalk.","marker":"[15]"},{"why":"UTMOS is the automated MOS predictor used to supplement human ratings in the speech evaluation.","marker":"[16]"},{"why":"Tacotron2 serves as one of the three TTS models trained on MultiTalk to demonstrate its usability as training data.","marker":"[17]"},{"why":"FastSpeech2 is the non-autoregressive TTS model used in the dataset-evaluation experiments.","marker":"[18]"}],"fun_headline_variants":["Agent trio crafts dialogue speech rivaling human-recorded data","LLM-writer, TTS-actor, critic-auditor: cheaper dialogue TTS data","Quarter the data, equal MOS: agent loop beats manual speech data","MultiTalk: agent-generated bilingual dialogue speech matches human sets","MultiTalk dataset released: agent-crafted dialogue speech rivals manual data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that using GPT-4o both as the script writer and as the evaluator of script naturalness and emotiveness does not bias the reported quality gains, and that the speech critic's feedback reflects genuine auditory quality rather than artifacts of the same model family.","fun_headline_variants_meta":{"raw":{"variants":["Agent trio crafts dialogue speech rivaling human-recorded data","LLM-writer, TTS-actor, critic-auditor: cheaper dialogue TTS data","Quarter the data, equal MOS: agent loop beats manual speech data","MultiTalk: agent-generated bilingual dialogue speech matches human sets","MultiTalk dataset released: agent-crafted dialogue speech rivals manual data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000602,"raw_usage":{"total_tokens":2769,"prompt_tokens":861,"completion_tokens":1908,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":1814}},"tokens_in":477,"tokens_out":1908,"duration_ms":11542,"temperature":1.0,"reasoning_tokens":1814,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:47:58.436722+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a panel of human raters score blinded dialogue scripts from the writer-only condition versus the two-loop critic condition on naturalness and emotiveness; if human ratings show no significant advantage for the critic-refined scripts, the claimed script-quality improvement collapses. Alternatively, train a TTS model on a random matched-size subset of DailyTalk and compare its MOS to a model trained on MultiTalk; if the subset-trained model matches or exceeds MultiTalk-trained MOS, the paper's central data-quality claim is falsified.","supporting_citations":[{"cited_title":"Dailytalk: Spoken dialogue dataset for conversational text-to-speech,","cited_arxiv_id":null,"evidence_quote":"DailyTalk is the primary dialogue-dataset baseline; its MOS and training setup anchor the claim that MultiTalk matches it with less than a quarter of the data."},{"cited_title":"The voice bank corpus: Design, collection and data analysis of a large regional accent speech database,","cited_arxiv_id":null,"evidence_quote":"VCTK supplies both a single-speaker TTS baseline and part of the character-pool audio used to craft character profiles."},{"cited_title":"The fisher corpus: a resource for the next generations of speech-to-text,","cited_arxiv_id":null,"evidence_quote":"The Fisher corpus serves as a prior multi-party conversational speech dataset with manual transcription, highlighting the cost problem MultiTalk aims to solve."},{"cited_title":"Common voice: A massively-multilingual speech corpus,","cited_arxiv_id":null,"evidence_quote":"Common Voice contributes audio to the character pool, supporting the diversity of voices in MultiTalk."},{"cited_title":"UTMOS: utokyo-sarulab system for voicemos challenge 2022,","cited_arxiv_id":null,"evidence_quote":"UTMOS is the automated MOS predictor used to supplement human ratings in the speech evaluation."},{"cited_title":"Natural TTS synthesis by conditioning wavenet on MEL spectro- gram predictions,","cited_arxiv_id":null,"evidence_quote":"Tacotron2 serves as one of the three TTS models trained on MultiTalk to demonstrate its usability as training data."},{"cited_title":"Fastspeech 2: Fast and high-quality end-to-end text to speech,","cited_arxiv_id":null,"evidence_quote":"FastSpeech2 is the non-autoregressive TTS model used in the dataset-evaluation experiments."}],"review_version":1}