REVIEW 3 major objections 6 minor 26 references
OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Large-scale synthetic spoken dialogue data, mixed sparingly with real recordings, lets a direct speech-to-speech model beat ASR-based systems on real-world conversations.
desk verdict The dataset is a real resource and the DailyTalk checkpoint is a genuine external result, but the SOTA claim is built on an unfair comparison and needs to be re-run before the synthetic-data story holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the heterogeneous feature fusion module Mix-Former, combined with a synthetic-data training recipe. For each dialogue turn, three frozen expert encoders produce temporally aligned features: Whisper for speech content, emotion2vec for emotional prosody, and BEATs for non-speech audio and music. Mix-Former applies a window-level Q-Former with a single learned query to each expert's features, computes a sigmoid weight per window per expert with linear layers, concatenates the weighted features, and projects them into a frozen Llama-3.1-8B-Instruct LLM trained only through LoRA adapters and the Q-Formers. The trained model predicts both response content and response style, namely the emotion label, which can be fed to a controllable TTS model to synthesize the reply. The dataset-generation pipeline is the other half of the mechanism: GPT-4 writes scenario-specific scripts seeded by AudioCaps and MusicCaps captions, CosyVoice-Instruct synthesizes speech with specified gender, pitch, speed, and emotion, and a diarization-plus-ASR check keeps word error rate under 5 percent.
What would settle it
Train OmniChat on ShareChatX alone and on the ShareChatX-plus-DailyTalk mix, then have human listeners score response naturalness and emotional appropriateness on held-out real conversations with spontaneous emotion and overlapping background audio; if listeners do not prefer the synthetic-trained model over a model trained only on recordings of equal size, or rate the synthetic corpus's emotional prosody as stereotyped, the transfer claim fails. Concretely, comparing emotion F1 on a spontaneous emotional speech test set against the reported 75.46 on DailyTalk would test whether synthetic emotion coverage transfers.
Extended reading notes
Core claim
OmniChat's central claim is that a spoken dialogue system can be made state-of-the-art on real conversations by pretraining on synthetic speech at scale and then lightly mixing in real data. Trained with synthetic ShareChatX and real DailyTalk at a 1:4 sampling ratio, the model reaches BLEU 4.95, METEOR 14.24, BERTScore 86.99, and emotion F1 75.46 on DailyTalk, surpassing direct audio models such as Qwen2-Audio and ASR-based systems such as FunAudioLLM on content and emotion. On the synthetic ShareChatX test sets, OmniChat outperforms the same baselines in all three scenarios, and ablation studies show that simply concatenating expert features can hurt while the Mix-Former weighted fusion recovers and improves performance. The paper further claims that as dataset size grows from 5K to 80K, direct speech-only input surpasses both text-only and ASR-augmented input, and that pre-training on large general emotion dialogues is needed before fine-tuning on audio or music scenarios.
Load-bearing premise
The approach assumes that synthetic dialogues produced by GPT-4 and CosyVoice faithfully represent real spoken conversation, including emotional fidelity and audio grounding, so that gains measured on ShareChatX transfer to real interactions.
Editorial extensions
If this is right
- A 20 percent synthetic-to-real sampling ratio is the reported sweet spot: lower ratios under-use synthetic diversity, while higher ratios drift from real conversational patterns.
- With enough training data, direct speech-input dialogue models overtake pipelines that feed ASR transcriptions, so the recommended design shifts toward end-to-end speech conditioning.
- Large-scale synthetic pre-training on general emotional dialogue is a necessary precondition for later fine-tuning on audio-event and music scenarios; small fine-tuning sets alone do not surpass it.
- Weighted, query-based fusion of content, emotion, and non-speech features beats naive feature concatenation, especially when music or audio events are present.
- ShareChatX provides benchmark splits for emotion, audio, and music dialogue, letting future systems be compared on scenario coverage rather than only on generic text metrics.
Reading between the lines
- If synthetic scaling transfers as reported, the remaining bottleneck for spoken dialogue is not recording hours but prompt and topic diversity plus TTS controllability, shifting data cost from collection to generation and verification.
- The optimal mixing ratio likely depends on how far the synthetic distribution sits from the target real distribution, so for high-stakes or spontaneous domains the paper's alpha of 0.2 would need to be re-estimated rather than assumed.
- A testable extension is to apply the same pipeline to lower-resource languages, where real spoken dialogue data is even scarcer and synthetic coverage may matter more.
- The paper's quality checks of word error rate, speaker diarization, and manual inspection do not measure whether synthetic emotional prosody sounds genuinely human, so an objective emotion-fidelity metric on synthesized speech would sharpen the transfer claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript introduces ShareChatX, a large-scale synthetically generated multi-turn spoken dialogue dataset with three subsets (emotion, audio, and music), and OmniChat, an LLM-based spoken dialogue system that fuses Whisper, emotion2vec, and BEATs features through a windowed Q-Former-based Mix-Former module. The authors report experiments on DailyTalk and on ShareChatX itself, claim state-of-the-art results on DailyTalk, and propose an optimal 20% synthetic-data sampling ratio (alpha=0.2) in mixed real/synthetic training.
Significance. The resource and the model are potentially valuable: a large, scenario-diverse synthetic spoken dialogue dataset could mitigate the scarcity and privacy issues of real spoken dialogue corpora, and the controlled alpha sweep is a sensible way to study synthetic/real data mixing. The paper also makes concrete architectural choices (frozen audio encoders and LLM, LoRA, windowed Q-Former) and promises release of data and code. However, the headline state-of-the-art claim currently rests on an uncontrolled comparison, and the complex-scenario results are evaluated in-distribution. These issues need to be addressed before the contribution can be accepted at its current strength.
major comments (3)
- [Section 5.2, Table 2; Appendix C.1] The claim of state-of-the-art performance on DailyTalk is not yet supported because OmniChat+Real Data is trained on DailyTalk (real samples are drawn with probability 1-alpha in the mixed training described in Appendix C.1), while all comparison systems in Table 2 are evaluated as released pre-trained models without fine-tuning. This comparison conflates target-domain fine-tuning with the value of synthetic data. Please fine-tune at least one strong baseline (e.g., Qwen2-Audio or FunAudioLLM) on the same DailyTalk data with the same training recipe, and also include an OmniChat model trained on real DailyTalk only (alpha=1.0) in Table 2. The sentence 'OmniChat demonstrated superior performance across all content-related metrics' is also contradicted by the table for OmniChat without real data: METEOR is 12.57 vs 13.89 for Qwen2-Audio, ROUGE-L is 12.63 vs 12.81, and GPT-eval is 3.96 vs 4.01.
- [Section 5.2, Table 3; Appendix C.2] The ShareChatX evaluations are in-distribution: the test dialogues are generated by the same GPT-4/CosyVoice pipeline used to construct the training data, and the baselines are not fine-tuned on ShareChatX. The large margins in Table 3 therefore partly reflect distribution matching rather than generalizable complex-scenario ability. To support the claim that synthetic data improves dialogue in diverse complex scenarios, please either fine-tune the baselines on the ShareChatX training splits, evaluate on an external real-world audio/music dialogue benchmark, or add human evaluation with objective audio-grounding checks.
- [Tables 2-5 and Figure 3] No error bars, multiple runs, or significance tests are reported. The DailyTalk test set contains only 220 samples, and several headline gaps are small (e.g., BERTScore 86.99 vs 86.14; GPT-eval 4.15 vs 4.01). The alpha=0.2 optimum in Table 4 and the scaling conclusions in Figure 3 are likewise stated without variance estimates, so it is unclear whether the observed differences are meaningful. Please add at least three random seeds with confidence intervals and appropriate statistical tests, such as bootstrap or paired tests on the test set.
minor comments (6)
- [Section 4.1, Eq. (1)] The dimension specified for Hs_i, He_i, Hb_i is unclear: the notation R^{[N x K/L] x D} appears to mix input expert dimension D with the Q-Former hidden dimension, and the bracketed exponent notation is ambiguous. Please clarify whether the output has N*K/L vectors or (N/L)*K vectors and specify the hidden dimension explicitly.
- [Section 5.3] The sentence 'the BLEU score of the direct speech model improves from 4.65 at 5K to 4.94 at 10K, while the text-based model lags behind with a BLEU score of 4.86' is confusing because 4.65 is below 4.86; please state which data scale and which comparison support the conclusion that speech models surpass text models at 5K-10K.
- [Section 4.1 and references] The model is referred to as 'Beat', but the cited work is BEATs (Chen et al., 2023); please use the correct model name and ensure the reference entry is complete.
- [Appendix A] The limitation section spells the model name as 'Omnichat' while the rest of the paper uses 'OmniChat'; please make the spelling consistent.
- [Appendix C.3] The GPT-eval description does not report the exact model version, decoding temperature, or number of evaluation passes, which limits reproducibility; please provide these details.
- [Table 2] The paper does not state whether the ASR-based baselines use reference transcriptions or predicted transcriptions on the DailyTalk test set; this should be specified because it affects the comparison between ASR-based and direct systems.
Circularity Check
DailyTalk SOTA is partly a fitted-input result: the synthetic/real mixing ratio α is chosen using DailyTalk performance and the same benchmark is then reported as state-of-the-art, while the remainder of the derivation is self-contained.
-
fitted input called prediction
[Section 5.4 (Table 4) with Section 5.2 (Table 2) and Appendix C.1]
"To explore this, we experimented with various sampling ratios, as shown in Table 4, to determine the optimal balance between synthetic and real data: ... Based on these findings, a sampling ratio of α = 0.2 provides the ideal balance, achieving optimal performance in real-world dialogue scenarios."
The only DailyTalk evaluation split described in the paper is the randomly selected 220-sample test set in Appendix C.1; no validation split is mentioned. Section 5.4 selects α by measuring performance on the DailyTalk dataset, declares α=0.2 the 'ideal balance', and the abstract plus Table 2 then report the real-data-mixed OmniChat as state-of-the-art on the same DailyTalk benchmark. The headline DailyTalk result is therefore the very quantity used to choose the reported configuration, i.e., a fitted hyperparameter renamed as a prediction, rather than an independent out-of-sample confirmation of that configuration.
full rationale
The strongest genuinely circular element is the α selection: the paper tunes the synthetic/real sampling ratio on DailyTalk and then presents DailyTalk numbers for the tuned model as the headline SOTA, with no described held-out validation split. This makes part of the central claim a fitted-input result. By contrast, the ShareChatX evaluations are standard i.i.d. tests on non-overlapping splits drawn from the same GPT-4/CosyVoice pipeline; that limits external generalization but is not a constructional circularity. The external DailyTalk benchmark does provide independent grounding for the model's real-world capability, which keeps the paper from being wholly self-referential. A separate and important confound, not itself circularity, is that OmniChat + Real Data is trained on real DailyTalk samples (Appendix C.1) while the Table 2 baselines are not reported as fine-tuned on DailyTalk; this undermines the synthetic-data attribution but is a benchmark-control problem rather than an equation-level reduction. Overall, the derivation of the model architecture and the ShareChatX dataset is self-contained; the partial circularity is concentrated in the DailyTalk 'ideal balance' and SOTA reporting.
Assumptions & free parameters
free parameters (3)
- sampling ratio alpha =
0.2
- Q-Former window size L =
17
- number of queries K =
1
assumptions (3)
- domain assumption LLM-generated scripts (GPT-4) are natural and contextually coherent enough to train a dialogue model.
- domain assumption CosyVoice-Instruct TTS preserves the specified emotion and style parameters accurately enough for the task.
- standard math The expert audio encoders (Whisper, Emotion2vec, BEATs) provide temporally aligned features that capture the needed information.
Cite this review
Pith. "Pith review of OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios." pith.science (2026). https://pith.science/paper/FAJCITHP
@misc{pith2026250101384,
author = {Pith},
title = {Pith review of: OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios},
year = {2026},
howpublished = {\url{https://pith.science/paper/FAJCITHP}},
note = {Machine review of arXiv:2501.01384}
}
read the original abstract
With the rapid development of large language models, researchers have created increasingly advanced spoken dialogue systems that can naturally converse with humans. However, these systems still struggle to handle the full complexity of real-world conversations, including audio events, musical contexts, and emotional expressions, mainly because current dialogue datasets are constrained in both scale and scenario diversity. In this paper, we propose leveraging synthetic data to enhance the dialogue models across diverse scenarios. We introduce ShareChatX, the first comprehensive, large-scale dataset for spoken dialogue that spans diverse scenarios. Based on this dataset, we introduce OmniChat, a multi-turn dialogue system with a heterogeneous feature fusion module, designed to optimize feature selection in different dialogue contexts. In addition, we explored critical aspects of training dialogue systems using synthetic data. Through comprehensive experimentation, we determined the ideal balance between synthetic and real data, achieving state-of-the-art results on the real-world dialogue dataset DailyTalk. We also highlight the crucial importance of synthetic data in tackling diverse, complex dialogue scenarios, especially those involving audio and music. For more details, please visit our demo page at \url{https://sharechatx.github.io/}.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
Musiclm: Generating music from text
Andrea Agostinelli, Timo Denk, Zalan Borsos, Jesse Engel, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Mauro Verzetti, Antoine Caillon, et al. Musiclm: Generating music from text. Junyi Ao, Yuancheng Wang, Xiaohai Tian, Dekun Chen, Jun Zhang, Lu Lu, Yuxuan Wang, Haizhou Li, and Zhizheng Wu. Sd-eval: A benchmark dataset for spoken dialogue ...
-
[4]
Qwen2-audio technical report.arXiv preprint arXiv:2407.10759,
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report.arXiv preprint arXiv:2407.10759,
-
[7]
20 Preprint version Kristina Jokinen and Michael McTear
URL https://arxiv.org/abs/2411.13577. 20 Preprint version Kristina Jokinen and Michael McTear. Spoken dialogue systems. Morgan & Claypool Publishers,
-
[10]
Dailytalk: Spoken dialogue dataset for conversational text-to-speech
Keon Lee, Kyumin Park, and Daeyoung Kim. Dailytalk: Spoken dialogue dataset for conversational text-to-speech. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,
work page 2023
-
[11]
Guan-Ting Lin, Cheng-Han Chiang, and Hung-yi Lee. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. arXiv preprint arXiv:2402.12786,
-
[12]
emotion2vec: Self-supervised pre-training for speech emotion representation
Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. emotion2vec: Self-supervised pre-training for speech emotion representation. arXiv preprint arXiv:2312.15185,
-
[13]
Expresso: A benchmark and analysis of discrete expressive speech resynthesis
Tu Anh Nguyen, Wei-Ning Hsu, Antony d’Avirro, Bowen Shi, Itai Gat, Maryam Fazel-Zarani, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, et al. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. arXiv preprint arXiv:2308.05725,
-
[16]
Meld: A multimodal multi-party dataset for emotion recognition in conversations
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508,
Show all 26 references
-
[18]
A neural network approach to context- sensitive generation of conversational responses
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and William B Dolan. A neural network approach to context- sensitive generation of conversational responses. In Proceedings of the 2015 Conference of th...
2015
-
[20]
Salmonn: Towards generic hearing abilities for large language models
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289,
-
[21]
Llama 2: Open founda- tion and fine-tuned chat models
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288,
-
[22]
E- chat: Emotion-sensitive spoken dialogue system with large language models
Hongfei Xue, Yuhao Liang, Bingshen Mu, Shiliang Zhang, Qian Chen, and Lei Xie. E- chat: Emotion-sensitive spoken dialogue system with large language models. arXiv preprint arXiv:2401.00475,
-
[23]
AIR-bench: Benchmarking large audio-language models via generative comprehension
Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. AIR-bench: Benchmarking large audio-language models via generative comprehension. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), ...
1979
-
[24]
URL https://aclanthology.org/2024.acl-long.109
Association for Compu- tational Linguistics. URL https://aclanthology.org/2024.acl-long.109. Dong Yu and Lin Deng. Automatic speech recognition, volume
2024
-
[26]
Bertscore: Evaluat- ing text generation with bert
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv:1904.09675,
1904 arXiv
-
[2002]
The cocktail fork problem: Three-stem audio separation for real-world soundtracks
Darius Petermann, Gordon Wichern, Zhong-Qiu Wang, and Jonathan Le Roux. The cocktail fork problem: Three-stem audio separation for real-world soundtracks. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 526–530. IEEE,
2022
-
[2004]
Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407,
-
[2008]
Emova: Empowering language models to see, hear and speak with vivid emotions
Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, et al. Emova: Empowering language models to see, hear and speak with vivid emotions. arXiv preprint arXiv:2409.18042,
-
[2009]
Audiocaps: Generating captions for audios in the wild
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1...
2019
-
[2015]
Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms
Tongyi SpeechTeam. Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms. arXiv preprint arXiv:2407.04051,
-
[2016]
Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities. arXiv preprint arXiv:2305.11000,
-
[2018]
End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions
Libo Qin, Wenbo Pan, Qiguang Chen, Lizi Liao, Zhou Yu, Yue Zhang, Wanxiang Che, and Min Li. End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5925–5941,
2023
-
[2020]
Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities
Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. arXiv preprint arXiv:2402.01831,
-
[2022]
Powerset multi-class cross entropy loss for neural speaker diariza- tion
Alexis Plaquet and Herv´e Bredin. Powerset multi-class cross entropy loss for neural speaker diariza- tion. In Proc. INTERSPEECH 2023,
2023
-
[2023]
Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919,
-
[2024]
The llama 3 herd of models
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.