Advancing Speech Language Models by Scaling Supervised Fine-Tuning with Over 60,000 Hours of Synthetic Speech Dialogue Data

Bajian Xiang; Qiang Niu; Shuaijiang Zhao; Tingwei Guo; Tongtang Wan; Wei Zou; Xiangang Li

REVIEW 5 major objections 5 minor 1 cited by

Advancing Speech Language Models by Scaling Supervised Fine-Tuning with Over 60,000 Hours of Synthetic Speech Dialogue Data

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 60,000-hour synthetic speech dialogue dataset can train a bilingual speech language model that outperforms comparable baselines.

desk verdict A large bilingual synthetic speech dialogue dataset is the real contribution; the model gains are real but measured only on synthetic speech, so the practical-interaction claim is overstated. read the letter →

arxiv 2412.01078 v2 pith:N3CYIV4V submitted 2024-12-02 cs.CL cs.AIcs.HC

Shuaijiang Zhao , Tingwei Guo , Bajian Xiang , Tongtang Wan , Qiang Niu , Wei Zou , Xiangang Li This is my paper · ORCID

classification cs.CLcs.AIcs.HC

keywords speechlanguagemodelsyntheticdialoguedatasetsupervisedfine-tuningbilingualinteractiontext-to-speechdataaugmentationspeech-textmodalityalignmentKE-OmniKe-SpeechChat

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that large-scale synthetic speech dialogue data can substitute for expensive real recordings when training a speech language model. It contributes a dataset construction pipeline—LLM rewriting of existing instruction text into spoken-style dialogues, TTS synthesis with privacy-protecting virtual voices, and CER/WER filtering—that produced 7 million Chinese and English conversations totalling over 60,000 hours. Trained on subsets of this data, the authors' KE-Omni model outperforms English-only baselines at comparable data scale on instruction-following, speech-text alignment, and speech quality. The practical stake is lowering the data cost and privacy barriers to building real-time speech assistants, especially for Chinese.

What carries the argument

The load-bearing mechanism is the synthetic data factory. Open-source instruction datasets are rewritten by LLMs (Qwen2.5-14B and 72B) through rewriting, filtering, and spoken-style post-processing to produce conversational text. A virtual voice library is built from premium WenetSpeech4TTS clips, using WavLM x-vectors to cluster clips by speaker, then averaging pairs of same-gender, same-rate voice embeddings to create privacy-protecting synthetic speakers that do not correspond to real individuals. CosyVoice synthesizes speech for these dialogues, AudioSeal watermarks the audio, and ASR-based CER/WER filtering drops low-quality clips. The KE-Omni model itself combines a frozen Whisper encoder with a trainable adapter compressing audio to 10 frames per second, a LLaMA-3.1-8B backbone, and a speech decoder with a duration predictor, a chunk-based autoregressive speech unit generator, and a HiFi-GAN vocoder.

What would settle it

Evaluate KE-Omni on a held-out set of real, spontaneous human conversations (with background noise, disfluencies, and overlapping speech) and measure WER/CER and human-rated naturalness; if scores drop materially relative to the synthetic chat-test, the claim that synthetic speech is a sufficient training proxy is overturned.

Watch

Extended reading notes

Core claim

The paper's central claim is that Ke-SpeechChat, a fully synthetic speech dialogue dataset built from rewritten text instructions, is high quality, and that a speech language model trained on it scales effectively. KE-Omni, trained on this dataset, achieves significantly better speech-to-text instruction-following scores than LLaMA-Omni, SpeechGPT, and Qwen2-Audio when trained on comparable data. Modality alignment, measured by CER/WER of spoken output, improves sharply as training data grows from 1,500 to 6,000 hours and saturates at the largest subsets, while speech quality (UTMOS) keeps improving up to the full 60,000 hours. The authors also report that KE-Omni performs competitively on the VoiceBench benchmark, with the caveat that its refusal style and spoken-form paragraph cues lower scores on AdvBench and IFEval.

Load-bearing premise

Speech synthesized by CosyVoice from LLM-rewritten text is a good enough stand-in for real human conversation to train a speech assistant, and the synthetic test sets fairly measure real-world ability.

Editorial extensions

If this is right

Speech language models can be trained from text-only instruction corpora plus TTS, removing the need for recorded conversational audio.
Data scale matters: modality alignment and speech quality improve as synthetic data grows, with alignment improving sharply beyond 4,000 hours.
Privacy can be built into the data pipeline via composite virtual voices and watermarking, addressing voice-misuse concerns.
Bilingual speech interaction research is extended beyond English-only systems such as LLaMA-Omni and SpeechGPT.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same pipeline likely transfers to other languages or domains where instruction text exists but conversational audio is scarce.
Because both training and test sets are synthetic and clean, real-world robustness to noise, disfluency, and overlapping speech remains an open question the paper does not settle; mixing synthetic with real dialogue data is a natural next step.
The fact that the L subset beats XL on CER/WER hints at a saturating or non-monotonic benefit of synthetic scale that deserves isolation from batch-size effects.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Ke-SpeechChat is the real contribution. The 60k-hour bilingual synthetic dialogue dataset, built by rewriting instruction data into spoken-style Q&A and synthesizing with CosyVoice across 42k virtual voices, is a valuable resource. The pipeline is sensible, the quality filters (ASR CER/WER, DNSMOS/UTMOS) are appropriate, and the downstream ASR/TTS checks give some evidence of quality. The virtual-speaker construction via weighted averaging of real embeddings is a nice privacy-preserving move.

KE-Omni itself is a fairly standard LLaMA-Omni-style architecture: Whisper encoder, LLaMA backbone, HuBERT units plus HiFi-GAN decoder. The scaling story is the main empirical finding: more synthetic data clearly helps S2TIF and speech quality, and CER/WER drop sharply from XS to S. That is useful for the field.

Now the soft spots. The strongest one: the chat-test used in Table 8 is also CosyVoice TTS, with the same speaker library design. So the big gains over LLaMA-Omni are measured in-distribution. The model may be learning TTS artifacts rather than general speech understanding. Without a benchmark on real human speech, the claim that KE-Omni advances practical real-time interaction is overstated. This doesn't invalidate the dataset, which is openly synthetic, but it should be labeled as a synthetic-speech evaluation, not a general speech interaction evaluation.

Second, the scaling experiment confounds data size with batch size. The authors themselves note that XL's higher CER/WER than L may be because the larger batch size in XL negatively affects performance. That's an uncontrolled variable, so the scaling conclusions are weaker than the headline suggests.

Third, no error bars or significance tests appear anywhere. Differences of 0.05 in S2TIF style scores are probably noise. Fourth, Whisper is used both to filter the synthetic audio and to compute the modality-alignment metrics, so there is a closed loop; a different ASR might give different numbers.

On citations, they cover the relevant prior work adequately, including LLaMA-Omni, Moshi, VoiceBench, and CosyVoice. I don't see a citation problem.

Bottom line: this is a dataset paper, and a useful one. The model is a competent baseline. The overclaim is in framing the work as advancing speech interaction when the evidence is confined to synthetic speech. A serious referee should ask for an evaluation on a real-speech benchmark, or at least a clear statement that results are for synthetic speech; deconfounding batch size; and error bars. I'd accept it for review, but with the expectation of significant revision.

Referee Report

5 major / 5 minor

Summary. The paper introduces Ke-SpeechChat, a large-scale synthetic bilingual speech dialogue dataset containing roughly 6.97 million Chinese and English conversations (about 60,000 hours) generated by rewriting open-source instruction data with Qwen LLMs and synthesizing speech with CosyVoice using virtual speaker embeddings derived from WenetSpeech4TTS. It also presents KE-Omni, an end-to-end speech-to-speech model built on a Whisper encoder, a LLaMA-3.1-8B backbone, and a duration-predictor/unit-generator/vocoder decoder. The authors report that KE-Omni outperforms existing speech language models on their own chat-test set across instruction-following, modality alignment, and speech quality metrics, and achieves competitive results on VoiceBench. The dataset quality is further assessed via DNSMOS/UTMOS, ASR fine-tuning on Whisper, and zero-shot TTS based on CosyVoice.

Significance. If the reported results are robust, the paper would provide a substantial resource for bilingual speech language model research, especially for Chinese, and a detailed pipeline for constructing synthetic speech interaction data at scale. The authors deserve credit for the detailed documentation of the data construction prompts (Appendix A), explicit quality-control stages (CER/WER filtering, DNSMOS-based voice selection), and the multi-task evaluation spanning ASR, TTS, speech interaction, and safety benchmarks. The dataset statistics and the plan for a two-stage training recipe are clearly presented. However, the central evidence for the model's real-world interaction capability is weakened by the in-distribution nature of the main test set and by the confounded scaling analysis. The significance of the contribution would be substantially strengthened by addressing these evaluation issues.

major comments (5)

[Section 5.1 and Section 5.5] The scaling study across the XS, S, M, L, and XL subsets confounds data volume with batch size. Section 5.1 states that 'in order to make the number of training steps as consistent as possible across all datasets, we adopted different batch sizes for different datasets in the second stage,' and Section 5.5 attributes the non-monotonic CER/WER (L better than XL) to the larger batch size in XL. As a result, the comparison does not isolate the effect of training data scale, and the claim that scaling data improves performance is not cleanly supported. Please either hold the batch size fixed across subsets, run a controlled ablation (e.g., XL with the L batch size), or explicitly frame the results as a joint function of data size and batch size. Reporting the batch sizes used for each subset would at least make the confound transparent.
[Section 5.2 and Table 8 / Table 10] The primary interaction benchmark, chat-test, is generated with the same CosyVoice pipeline and the same style of virtual-speaker audio as the training data. Training and evaluation are therefore in-distribution with respect to the TTS system and voice library. The large margins over baselines in Table 8 may partly reflect adaptation to CosyVoice artifacts rather than generalizable speech interaction ability, especially since the baseline models were not trained on CosyVoice audio. The authors should evaluate KE-Omni on real human speech (e.g., recorded human instructions on the same tasks or a public spoken dialogue benchmark) and, for the VoiceBench results in Table 9, report the real and synthetic instruction subsets separately instead of pooled scores. Section 6 should explicitly acknowledge the in-distribution nature of the main test set.
[Tables 6-9 and Section 5.5] All reported results are single runs without error bars, confidence intervals, or significance tests. Several headline comparisons are close: e.g., L vs XL in Chinese CER (5.03 vs 5.16), L vs XL in UTMOS (3.39 vs 3.43), and KE-Omni-L vs KE-Omni-XL on AlpacaEval (3.74 vs 3.78). Without variance estimates, it is impossible to judge whether these differences are meaningful. Please provide results over at least three random seeds (or repeated evaluations with different holdout subsets) and apply appropriate statistical tests for the main comparisons in Table 8 and Table 9.
[Section 5.5, Table 8] The claim that 'KE-Omni achieves significantly better performance than baseline systems when trained on datasets of comparable size' is not substantiated by a controlled comparison. The baseline models (LLaMA-Omni, SpeechGPT, Qwen2-Audio) were trained on their own datasets, not on the Ke-SpeechChat subsets, so the comparison confounds model architecture, training data, and training procedure. To support this claim, the authors would need to train the baseline architectures on the same Ke-SpeechChat subsets, or at least train them on the same amount of data drawn from the same distribution. Otherwise, the claim should be rephrased as 'KE-Omni outperforms off-the-shelf baselines on our test set,' which is a weaker but accurate statement.
[Section 3.2.3 and Section 5.3] The quality-assurance step in dataset construction uses Whisper-family ASR models to compute CER/WER and filter dialogues, and the subsequent modality-alignment evaluation of KE-Omni also uses Whisper-large-v3 for transcription. This creates a closed-loop risk: the training data is selected to be easily transcribable by Whisper, which may inflate the measured CER/WER of the final model. The authors should evaluate the generated speech with a different ASR system (e.g., an independently trained model) and discuss the potential bias. At minimum, they should acknowledge this circularity in Section 6.

minor comments (5)

[Abstract] The sentence '...totaling over 60,000 hours, This contributes significantly...' is a run-on with a missing period; 'This' should begin a new sentence and likely refer to 'this dataset' or 'this work'.
[Section 3.3.2] The text says 'including 40,000 users and 2 agents,' but Table 1 lists 21,000 male users and 21,000 female users, totaling 42,000 users. Please correct the number to 42,000 and clarify whether the same user speaker set is used for both languages.
[Section 5.1] The sentence 'KE-Omni use LLaMA-3.1-8B-Instruct(Fang et al., 2024)' cites Fang et al. (LLaMA-Omni) for the LLaMA-3.1 model; the correct reference is Dubey et al. (2024), the LLaMA 3 herd paper. Also, 'use' should be 'uses'.
[Section 5.5] The phrase 'KE-Omni outperforms to other baseline systems' is grammatically incorrect; it should be 'outperforms other baseline systems.'
[Figure 2 and Section 3.2.1] The condition 'Max Count ≥ 5 × ⌊n/10⌋' appears in the figure caption but is not defined in the main text; please introduce the notation for ⌊n/10⌋ and explain the threshold clearly in Section 3.2.1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports empirical dataset construction and model training, and its central claims are independently anchored by external benchmarks; the synthetic in-distribution evaluation is a validity limitation, not a circular derivation.

full rationale

This paper does not present a derivation that reduces to its inputs; it is an empirical systems paper. Ke-SpeechChat is built by LLM rewriting (Qwen2.5), TTS synthesis (CosyVoice), and ASR-based CER/WER filtering (Whisper), and KE-Omni is trained on it. The headline results in Table 8 are measured on chat-test, which is synthesized with the same CosyVoice pipeline (Section 5.2), so the evaluation is partly in-distribution; however, this is a validity/external-transfer risk, not a circularity, because the comparison is a measurement rather than a construction, and the paper also reports external benchmarks (AISHELL-1 and LibriSpeech in Table 6, SeedTTS in Table 7, VoiceBench in Table 9) that do not depend on the paper's fitted values. Self-citations (BELLE, DiDiSpeech, ChatHome references and the Belle-whisper model) are sources or tools, not load-bearing evidence for the central claims. The limitation section (Section 6) explicitly concedes that the synthetic speech is clean, noiseless, and single-turn, which supports the skeptical reading that real-speech generalization is untested, but that is a correctness or scope concern rather than a circular derivation. No equation or fitted parameter is renamed as a prediction, and no uniqueness claim is imported from the authors' own prior work.

Assumptions & free parameters 6 free parameters · 4 assumptions · 1 invented entities

The central claims rest on several hand-chosen thresholds for data construction and on domain assumptions about synthetic data quality. There are no free parameters in a mathematical derivation sense, but the data pipeline is governed by these choices, and the scaling experiment is confounded by batch size choices.

free parameters (6)

CER/WER filtering thresholds = 5% (Chinese CER), 10% (English WER)
Used in Section 3.2.3 to drop low-quality synthetic dialogues, directly shaping the dataset.
DNSMOS premium clip threshold = 4.0
Section 3.2.1 selects only clips with DNSMOS at least 4.0 for the voice library.
Speaker similarity threshold = 0.97
Section 3.2.1 uses cosine similarity above 0.97 to group clips of the same speaker.
Minimum clips per recording and pair count = n >= 10, pairs >= 5 * floor(n/10)
Heuristics in Section 3.2.1 to identify a main speaker in a recording.
Response length cap = 100 words
Section 3.1.3 instructs the LLM to generate responses under 100 words, constraining dataset content.
Batch size per subset = Varied by subset
Section 5.1 changes batch size to keep training steps consistent, but this confounds the scaling results (XL batch size blamed for worse CER).

assumptions (4)

domain assumption Open-source instruction datasets (BELLE, LaMini, IndustryInstruction) are a valid source for generating spoken-style dialogues.
Section 3.1 uses these datasets as the raw material for rewriting.
domain assumption TTS-generated audio with virtual voices is acoustically suitable for training a speech language model.
Section 3.2 generates all training audio with CosyVoice from virtual voice embeddings.
domain assumption Whisper-based CER/WER filtering reliably measures synthetic speech quality.
Section 3.2.3 uses Whisper transcriptions to drop high-error dialogues, assuming low error implies high quality.
domain assumption Qwen2.5 LLM rewriting preserves task meaning while converting to spoken style.
Sections 3.1.1 to 3.1.3 rely on the LLM to rewrite instructions and generate responses without verification of semantic equivalence.

invented entities (1)

Virtual speaker embeddings
purpose: To create a large pool of privacy-protecting synthetic voices for TTS synthesis.
Created by weighted averaging of real speaker embeddings (Section 3.2.1). No external validation that the voices are truly unidentifiable or distinct; the claim of privacy is an assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Advancing Speech Language Models by Scaling Supervised Fine-Tuning with Over 60,000 Hours of Synthetic Speech Dialogue Data." pith.science (2026). https://pith.science/paper/N3CYIV4V

@misc{pith2026241201078,
  author       = {Pith},
  title        = {Pith review of: Advancing Speech Language Models by Scaling Supervised Fine-Tuning with Over 60,000 Hours of Synthetic Speech Dialogue Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N3CYIV4V}},
  note         = {Machine review of arXiv:2412.01078}
}

read the original abstract

The GPT-4o represents a significant milestone in enabling real-time interaction with large language models (LLMs) through speech, its remarkable low latency and high fluency not only capture attention but also stimulate research interest in the field. This real-time speech interaction is particularly valuable in scenarios requiring rapid feedback and immediate responses, dramatically enhancing user experience. However, there is a notable lack of research focused on real-time large speech language models, particularly for Chinese. In this work, we present KE-Omni, a seamless large speech language model built upon Ke-SpeechChat, a large-scale high-quality synthetic speech interaction dataset consisting of 7 million Chinese and English conversations, featuring 42,002 speakers, and totaling over 60,000 hours, This contributes significantly to the advancement of research and development in this field. The demos can be accessed at \url{https://huggingface.co/spaces/KE-Team/KE-Omni}.

Figures

Figures reproduced from arXiv: 2412.01078 by the authors.

**Figure 2.** The construction of speech dialogue data. (A) Construction of real speakers. (B) Construction of virtual [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Model architecture of KE-Omni. speed, we utilized a chunk-based autoregressive approach, predicting speech units chunk by chunk. Given chunk size C and the length of target speech unit sequence T, the embeddings of ith text token and the jth speech unit are concatenated as input, where j = i − C and i ∈ [C, T]. Zero-embeddings of speech units are used at the first chunk. To ensure the quality of speech unit gener… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Evaluation methodology for large speech language model, focusing on three key capabilities: Speech-to-Text Instruction-Following(S2TIF), Modality Alignment and Speech Quality. To explore the impact of data size on model performance, we trained KE-Omni models on vario…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

DialogueAgents: A Hybrid Agent-Based Speech Synthesis Framework for Multi-Party Dialogue
cs.CL 2025-04 conditional novelty 5.0 of 10

A three-agent script-writer, synthesizer, and critic pipeline generates a bilingual multi-party speech dataset whose quality matches manually assembled datasets in TTS training benchmarks.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages · cited by 1 Pith paper

[1]

this" or

Clear References: Ensure all necessary context and background are included in the question. Avoid using vague terms like "this" or "that"

work page
[2]

arXiv preprint arXiv:2407.10759

Qwen2-audio technical report. arXiv preprint arXiv:2407.10759. [Défossez et al.2024] Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text founda- tion model for real-time dialogue. arXiv preprint arXiv:2410.00037. [Du et al.2024] Zhihao Du, Qian Chen, Shili...

arXiv 2024
[3]

Suitable for AI: The question should be appropriate for asking an AI but should not explicitly mention AI

work page
[4]

arXiv preprint arXiv:2407.05361

Emilia: An extensive, multilingual, and di- verse speech dataset for large-scale speech genera- tion. arXiv preprint arXiv:2407.05361. 11 [Hsu et al.2021] Wei-Ning Hsu, Benjamin Bolte, Yao- Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learn- ing by masked prediction o...

arXiv 2021
[5]

[OpenAI2024] OpenAI

Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model bench- mark. [OpenAI2024] OpenAI. 2024. Gpt-4o:the new flagship model that can reason across audio, vision, and text in real time. Blog post. [Pratap et al.2020] Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. Mls: A large-scale mu...

arXiv 2024
[6]

arXiv preprint arXiv:2410.11190

Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities. arXiv preprint arXiv:2410.11190. [Yang et al.2024] Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yi- chong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. 2024. Air-bench: Benchmarking large audio-language models via generative comprehen- sion. arX...

arXiv 2024
[7]

[Zhang et al.2022] Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng

Anygpt: Unified multimodal llm with discrete sequence modeling. [Zhang et al.2022] Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng. 2022. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. [Zhang et al.2023] Dong Zhang, Shimin Li, Xin Z...

arXiv 2022
[9]

Verbal Resolution: The question should be answerable verbally and should not require generating articles, images, or other content

work page

Show all 13 references

[11]

Single Question: Include only one question, and it does not need to fully incorporate the content of the information piece
[12]

Privacy: Do not include any private information such as phone numbers, websites, or social media handles
[13]

is_suitable_for_speech

Difficulty Level: For factual questions, aim for topics that are common and relatable, avoiding obscure or niche subjects. For emotional questions, provide specific details. ##Information Piece : {instruction} Please output only the question you have formulated: Prompt for Fil...
[2021]

In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 6968–6972

Didispeech: A large scale mandarin speech corpus. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 6968–6972. IEEE. [He et al.2024] Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Li- wei L...

2021
[2024]

arXiv preprint arXiv:2410.17196

V oicebench: Benchmarking llm-based voice assistants. arXiv preprint arXiv:2410.17196. [Chu et al.2024] Yunfei Chu, Jin Xu, Qian Yang, Hao- jie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al

2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

this" or

Clear References: Ensure all necessary context and background are included in the question. Avoid using vague terms like "this" or "that"

work page

[2] [2]

arXiv preprint arXiv:2407.10759

Qwen2-audio technical report. arXiv preprint arXiv:2407.10759. [Défossez et al.2024] Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text founda- tion model for real-time dialogue. arXiv preprint arXiv:2410.00037. [Du et al.2024] Zhihao Du, Qian Chen, Shili...

arXiv 2024

[3] [3]

Suitable for AI: The question should be appropriate for asking an AI but should not explicitly mention AI

work page

[4] [4]

arXiv preprint arXiv:2407.05361

Emilia: An extensive, multilingual, and di- verse speech dataset for large-scale speech genera- tion. arXiv preprint arXiv:2407.05361. 11 [Hsu et al.2021] Wei-Ning Hsu, Benjamin Bolte, Yao- Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learn- ing by masked prediction o...

arXiv 2021

[5] [5]

[OpenAI2024] OpenAI

Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model bench- mark. [OpenAI2024] OpenAI. 2024. Gpt-4o:the new flagship model that can reason across audio, vision, and text in real time. Blog post. [Pratap et al.2020] Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. Mls: A large-scale mu...

arXiv 2024

[6] [6]

arXiv preprint arXiv:2410.11190

Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities. arXiv preprint arXiv:2410.11190. [Yang et al.2024] Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yi- chong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. 2024. Air-bench: Benchmarking large audio-language models via generative comprehen- sion. arX...

arXiv 2024

[7] [7]

[Zhang et al.2022] Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng

Anygpt: Unified multimodal llm with discrete sequence modeling. [Zhang et al.2022] Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng. 2022. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. [Zhang et al.2023] Dong Zhang, Shimin Li, Xin Z...

arXiv 2022

[8] [9]

Verbal Resolution: The question should be answerable verbally and should not require generating articles, images, or other content

work page