Pith. sign in

REVIEW 4 major objections 6 minor 39 references

SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SOVA-Bench is a benchmark for generative speech LLMs that measures knowledge, recognition, linguistic and paralinguistic understanding, and semantic and acoustic generation, and reports that current speech-native models trail a…

desk verdict Useful benchmark scaffold and honest model comparison, but the Whisper-filtered TTS pipeline biases the headline comparison and the incomplete table undercuts 'comprehensive'. read the letter →

arxiv 2506.02457 v1 pith:JX6BK6WG submitted 2025-06-03 cs.SD eess.AS

Yixuan Hou , Heyang Liu , Yuhao Wang , Ziyang Cheng , Ronghua Wu , Qunshan Gu , Yanfeng Wang , Yu Wang This is my paper
classification cs.SDeess.AS
keywords speechlargelanguagemodelvoiceassistantbenchmarkgenerationqualityparalinguisticunderstandingspokenquestionansweringzero-shotTTSsynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SOVA-Bench is an evaluation benchmark for speech large language models and voice assistants, built to measure not just whether a model understands speech but also whether the voice it generates is accurate, fluent, and natural. It assembles eight tasks from six source datasets: general-knowledge QA, speech recognition, spoken question answering in two formats, emotion/gender/age recognition, and generation consistency, semantic quality, and acoustic quality. The authors run six open speech-native assistants and a cascade baseline of a speech recognizer followed by a text LLM. The central finding is that no model wins on every axis: larger-backbone models do better on knowledge and semantics, acoustic quality depends on tokenization and vocoder design, and the cascade beats every speech-native model by a wide margin on knowledge and spoken understanding. The paper offers SOVA-Bench as a common standard for comparing voice interaction systems on both semantic and acoustic grounds.

What carries the argument

The machinery is the task taxonomy and the audio-synthesis pipeline. The taxonomy splits voice-assistant ability into five properties, each assigned dedicated subsets so the benchmark can attribute a model's failure to knowledge, perception, paralinguistic perception, or generation. For datasets that exist only as text (TriviaQA and Alpaca), the pipeline synthesizes speech with CosyVoice using a multi-speaker prompt set from Common Voice, then filters the audio through Whisper-large-v3 so only clips that transcribe accurately are kept. Generation is scored three ways: WER between the generated speech's transcription and the text response for consistency, GPT-based evaluation for semantic quality, and UTMOSv2 for acoustic quality. This combination is what lets SOVA-Bench separate 'what the model says' from 'how the model sounds.'

What would settle it

Re-run SOVA-Bench's knowledge and generation subsets with real human recordings (same prompts, same speaker distribution) instead of CosyVoice audio filtered by Whisper-large-v3; if model rankings or the gap with the cascade shift substantially, the synthetic-speech filter, not the measured abilities, is driving the result.

Watch

Extended reading notes

Core claim

The paper's central claim is that a voice assistant benchmark should cover five properties—general knowledge, speech recognition, linguistic understanding, paralinguistic understanding, and speech generation—and that existing benchmarks miss the generation side. SOVA-Bench operationalizes this with eight subsets: TriviaQA questions synthesized to speech for knowledge; LibriSpeech for recognition; LibriSQA and Spoken SQuAD for linguistic understanding; IEMOCAP and Common Voice for emotion, gender, and age; and Alpaca prompts synthesized to speech for generation, scored for text-speech consistency by WER, semantic quality by GPT-based scoring, and acoustic quality by UTMOSv2. On these tasks, the paper reports that current speech-native LLMs degrade relative to their text counterparts: the best speech-native model reaches 27.27% on knowledge QA while the cascade reaches 58.98%, and the gap is even larger on spoken QA. Generation results are mixed, with no model strong on all three generation scores. The paper concludes that the main limitations of current voice interaction models are limited input modalities, inconsistent instruction following, a performance drop when text capabilities are extended to speech, and unsatisfying output speech quality.

Load-bearing premise

The knowledge and generation scores assume that CosyVoice-synthesized speech that passes a Whisper transcription check represents ordinary conversational voice well enough that conclusions carry over to real users.

Editorial extensions

If this is right

  • If SOVA-Bench is adopted, voice-assistant progress will be measured on both semantic accuracy and acoustic quality rather than on transcription alone.
  • Speech-native models that want to compete with cascades must close the knowledge and spoken-QA gap, since even a 7B or 8B speech LLM trails Whisper-plus-GPT-4o-mini by roughly 30 to 50 points on spoken QA.
  • Paralinguistic tasks show that instruction following, not just perception, is a bottleneck: low answer rates hide higher conditioned accuracy.
  • Generation consistency and semantic quality are partly determined by the LLM backbone and decoder design, so future work should target tokenization and training schemes rather than only model scale.
  • The benchmark's eight-task structure offers a common reporting format across future voice assistants.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because knowledge and generation audio is synthesized by CosyVoice and filtered by Whisper, the rankings may transfer best to TTS-like audio; a version using spontaneous human speech could change the knowledge and acoustic scores.
  • The consistency metric uses ASR on generated speech, so a model that produces expressive or accented speech could be penalized for ASR errors; human listening tests would be needed to confirm perceptual quality.
  • SOVA-Bench is single-turn; multi-turn context, barge-in, and latency are central to real voice assistants and may reveal different failure modes.
  • The current acoustic scorer is a predictor, not a listener; pairing it with subjective ratings would make the acoustic axis more decisive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SOVA-Bench, an evaluation framework for generative speech LLMs and voice assistants. It aims to compare models on eight tasks spanning general knowledge (TriviaQA), speech recognition (LibriSpeech), linguistic understanding (LibriSQA and Spoken SQuAD), paralinguistic understanding (IEMOCAP and Common Voice), and speech generation (Alpaca), measuring consistency, semantic quality, and acoustic quality. The authors evaluate Mini-Omni, Mini-Omni2, Moshi, LLaMA-Omni, Freeze-Omni, GLM-4-Voice, and a Whisper-large-v3 + GPT-4o-mini cascade, and report that speech-native models underperform the cascade on knowledge and understanding while generation quality varies widely across models. The paper concludes by listing limitations of current speech LLMs.

Significance. If its measurement pipeline is sound, SOVA-Bench fills a real gap: most existing benchmarks measure understanding or transcribe speech output to text, whereas SOVA-Bench attempts to score acoustic and paralinguistic generation directly. The inclusion of a cascade baseline is a useful reference point, and the comparison of speech-native systems against a strong ASR+LLM pipeline is empirically valuable. Credit is due for covering eight tasks and for using external scoring tools rather than fitted parameters, so there is no circularity in the sense of a derivation that assumes its own conclusion. However, the validity of the headline comparison is currently impaired by the Whisper-based data-selection rule, by large amounts of missing data in Table 4, by borrowed scores from other papers, and by underspecified evaluation protocols. The benchmark's usefulness as a standard therefore depends on the revisions described below.

major comments (4)
  1. [§3.2, Figure 2, Table 4] The construction of the Knowledge and Generation subsets embeds the reference system into the data-selection rule. After CosyVoice synthesis, instances are kept only when Whisper-large-v3 transcribes them with 'high accuracy'; the cascade baseline in §4.1 is exactly Whisper-large-v3 followed by GPT-4o-mini. The filter therefore selects inputs on which the baseline's ASR component already succeeds and discards the hard, noisy, accented, or disfluent utterances on which it would fail, biasing the comparison in favor of the cascade and also favoring LLaMA-Omni, whose speech encoder is Whisper-large-v3. The same filtering removes natural conversational variability, so the Knowledge and Generation scores are not shown to transfer to real user speech. Please report the accuracy threshold and the fraction of instances discarded, evaluate the cascade (and ideally all models) on the unfiltered sets, and either use a filter independent of the baseline ASR or add a natural-speech condition.
  2. [Table 4, §4.2] The reported results are incomplete for the central cross-model comparison. Mini-Omni and Mini-Omni2 lack LibriSQA and Spoken SQuAD scores; Moshi lacks all paralinguistic scores; LLaMA-Omni lacks recognition scores; the cascade lacks paralinguistic, consistency, and acoustic scores. In addition, Moshi and GLM-4-Voice were evaluated on only 500 randomly sampled instances per subset, and no confidence intervals or significance tests are provided. With this amount of missing data, the 'same evaluation framework' claim and the conclusions in §4.3 are not fully supported. Please complete the matrix or explicitly restrict each comparative claim to the models and tasks that were actually measured, and report sample sizes and uncertainty.
  3. [§4.2 (recognition rows, Table 4 note)] The recognition scores for Freeze-Omni and GLM-4-Voice are borrowed from their original papers rather than measured under the SOVA-Bench protocol. Because those papers use different prompts, decoding settings, audio pre-processing, and possibly different WER conventions, these entries are not directly comparable to the other rows in Table 4. The manuscript should either run these models with the same protocol or clearly designate the borrowed entries as external and exclude them from the comparative statements.
  4. [§3.2, Table 2] The evaluation protocol for the GPT-based metrics is underspecified. For Knowledge, Spoken SQuAD, and the paralinguistic tasks, 'GPTEval Accuracy' is reported, but the text does not define the judge model, the prompt template, the output parsing, or how refusals and malformed answers are scored. Similarly, 'Consistency' is measured as WER between the text response and the model's speech output, but the ASR used to transcribe the generated speech is not stated. If Whisper-large-v3 is used again, the consistency scores inherit its recognition errors and will penalize models with highly synthetic voices. Please specify all scoring pipelines and release the prompts and scoring code so that the benchmark is reproducible.
minor comments (6)
  1. [Abstract] The abstract contains 'Speech cOnversationalVoiceAssistantBenchmark' and 'comprehension comparison'; these should read 'Conversational' and 'comprehensive comparison'.
  2. [Table 4] Several entries are typeset with missing separators, e.g., '1.683.66' and '3.29/7.403.24', and some model names in Table 3 appear as 'GLM-4-V oice' due to line-breaking; please fix the formatting.
  3. [Section 3.2] The 'high accuracy' threshold used for Whisper filtering is not defined; the exact criterion (e.g., word error rate or character accuracy cutoff) and the resulting number of retained instances per subset should be stated.
  4. [Section 4.1] The 2.5-second blank audio prepended for Moshi is motivated, but there is no control experiment showing that this duration does not affect scores on other tasks; please provide a sensitivity check or further justification.
  5. [Section 3.2] The paper claims that the synthesis-and-filter pipeline 'ensures the diversity of speakers and acoustic environments,' but Figure 2 is a schematic; please provide quantitative statistics on speaker counts, acoustic conditions, and the distribution of discarded examples.
  6. [General] The paper does not state whether the benchmark data, prompts, and evaluation scripts will be released; for a benchmark contribution this is essential for the framework to be adopted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SOVA-Bench is an externally sourced evaluation benchmark with no fitted parameters or self-citation chain that reduces its claims to its inputs by construction.

full rationale

SOVA-Bench is a benchmark paper rather than a derivation. The measured quantities are accuracies, WERs, and external model scores computed on datasets sourced from independent corpora (TriviaQA, LibriSpeech, LibriSQA, Spoken SQuAD, IEMOCAP, Common Voice, and Alpaca), with scoring performed by external tools (Whisper-large-v3, GPT-based evaluation, and UTMOSv2). No parameter is fitted from the benchmark results and then renamed as a prediction; no central claim is defined in terms of the evidence it is supposed to provide. The Whisper-large-v3 accuracy filter in Section 3.2 does select synthesized items transcribed correctly by Whisper, and the cascade baseline in Section 4.1 uses the same ASR model plus GPT-4o-mini. That is a legitimate external-validity or selection-bias concern about the benchmark's representativeness, but it is not circularity: the filter does not by construction force the cascade's scores, the cascade is an explicit comparison system rather than a fitted parameter, and the benchmark's comparisons among speech LLMs are not equivalent to the filtering rule. The paper also borrows some model results from prior papers, but those are ordinary citations of external evaluations, not load-bearing self-citations that define the benchmark's content. No circular step can be exhibited, so the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The benchmark's validity rests on several domain assumptions rather than fitted parameters. The main assumptions are that CosyVoice-synthesized and Whisper-filtered audio represents natural speech, that GPT-based accuracy scoring and GPTScore are valid proxies, that UTMOSv2 tracks human acoustic quality judgments, and that the chosen datasets and label mappings define the target abilities. No invented entities and no free parameters are needed for the benchmark itself.

assumptions (6)
  • domain assumption CosyVoice zero-shot TTS produces speech that is representative of natural conversational speech for evaluation purposes.
    Used in Section 3.2 to create speech audio for TriviaQA and Alpaca; if TTS audio differs systematically from real user speech, benchmark results may not generalize.
  • domain assumption Whisper-large-v3 filtering retains only high-quality transcriptions and does not bias the benchmark toward easier samples.
    Section 3.2 and Figure 2; filtering on ASR accuracy may discard harder or more naturalistic audio.
  • domain assumption GPT-4 based evaluation (GPTEval accuracy and GPTScore) is a valid proxy for human assessment.
    Used in Table 2 for Knowledge, LibriSQA, Spoken SQuAD, emotion/gender/age tasks, and semantic generation; no human agreement or calibration study is reported.
  • domain assumption UTMOSv2 MOS predictions reflect human-perceived acoustic quality.
    Used for acoustic generation scores in Table 2; predictor-based MOS is approximate and may not capture all dimensions of naturalness.
  • domain assumption IEMOCAP emotion labels can be merged into a 5-class task and Common Voice gender and age annotations are reliable ground truth.
    Section 3.2 paralinguistic tasks; label merging and annotation reliability directly affect measured accuracies.
  • domain assumption The selected source datasets (LibriSpeech, LibriSQA, Spoken SQuAD, TriviaQA, Alpaca) cover the target voice assistant abilities.
    Section 3.2; benchmark validity depends on task coverage and difficulty being representative of real voice assistant use.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant." pith.science (2026). https://pith.science/paper/JX6BK6WG

@misc{pith2026250602457,
  author       = {Pith},
  title        = {Pith review of: SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JX6BK6WG}},
  note         = {Machine review of arXiv:2506.02457}
}
read the original abstract

Thanks to the steady progress of large language models (LLMs), speech encoding algorithms and vocoder structure, recent advancements have enabled generating speech response directly from a user instruction. However, benchmarking the generated speech quality has been a neglected but critical issue, considering the shift from the pursuit of semantic accuracy to vivid and spontaneous speech flow. Previous evaluation focused on the speech-understanding ability, lacking a quantification of acoustic quality. In this paper, we propose Speech cOnversational Voice Assistant Benchmark (SOVA-Bench), providing a comprehension comparison of the general knowledge, speech recognition and understanding, along with both semantic and acoustic generative ability between available speech LLMs. To the best of our knowledge, SOVA-Bench is one of the most systematic evaluation frameworks for speech LLMs, inspiring the direction of voice interaction systems.

Figures

Figures reproduced from arXiv: 2506.02457 by the authors.

Figure 1
Figure 1. Properties for an ideal voice assistant. The color of the generated response corresponds to distinct aspects of the user’s query. For example, if the model receives emotions of sadness and fear, a comforting tone should usually be chosen. following, robustness and safety alignment [9]. However, the above benchmarks either only measure the model’s ability to produce text responses or transform the speech response via… view at source ↗
Figure 2
Figure 2. Creation pipeline for dataset without speech modality. A multi-speaker prompt set is adopted for zero-shot TTS. ers and acoustic environments, thereby more robustly evaluat￾ing the performance. 4. Evaluation Experiment 4.1. Evaluated Models We have tested various widely used voice assistants on SOVA￾bench, as shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance for paralinguistic understanding. 4.2. Main Results The overall performance of tested speech LLMs is presented in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 18 canonical work pages

  1. [1]

    Recent advancements in large language models (LLMs) have led to remarkable breakthroughs in speech LLMs and voice assistants

    Introduction Communication through speech outperforms text interaction for its convenience and efficiency. Recent advancements in large language models (LLMs) have led to remarkable breakthroughs in speech LLMs and voice assistants. Starting from an LLM trained on tremendous textual corpora, these models perceive speech using an encoder with adaptors, gen...

    arXiv 2025

  2. [2]

    Speech LLM Speech LLM extends the understanding capability to speech flow, performing modality alignment between speech and text via an encoder with adaptors

    Related Works 2.1. Speech LLM Speech LLM extends the understanding capability to speech flow, performing modality alignment between speech and text via an encoder with adaptors. Earlier work focused on gen- erating textual responses from speech instructions, optionally adopting a TTS module for a cascaded speech-in-speech-out paradigm, such as SALMONN [10...

    work page

  3. [3]

    SOV A-Bench As shown in Table 2, SOV A-Bench aims to comprehensively evaluate and compare speech LLMs’ abilities to perceive in- structions and generate responses both in speech formats, there- fore contributing to the evolution of voice assistants. 3.1. V oice Assistant Properties V oice assistants are intelligent systems designed to enhance human-comput...

    work page

  4. [4]

    Evaluation Experiment 4.1. Evaluated Models We have tested various widely used voice assistants on SOV A- bench, as shown in Table 3, including: Mini-Omni [2], Mini- Omni2 [32], LLaMA-Omni [3], Freeze-Omni [18], Moshi [4], and GLM-4-V oice [33]. Additionally, we developed a cascade model for comparative evaluation, which combines Whisper-large-v3 as the A...

    work page arXiv

  5. [5]

    Conclusion In this paper, we introduce a comprehensive benchmark, SOV A- Bench, for evaluating generative speech LLMs from both se- mantic and acoustic perspectives. Evaluation experiments on novel speech-interactive models reveal variations in the qual- ity of generated speech responses and a substantial degrada- tion in performance compared to the casca...

    work page

  6. [6]

    Acknowledgements This work was supported by the Shanghai Jiao Tong University - Ant Group Intelligent Multimedia Joint Laboratory

    work page

  7. [7]

    GPT-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radfordet al., “GPT-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    arXiv 2024

  8. [8]

    Mini-Omni: Language models can hear, talk while thinking in streaming,

    Z. Xie and C. Wu, “Mini-Omni: Language models can hear, talk while thinking in streaming,”arXiv preprint arXiv:2408.16725, 2024

    arXiv 2024

Show all 39 references
  1. [9]

    LLaMA-Omni: Seamless speech interaction with large language models,

    Q. Fang, S. Guo, Y . Zhou, Z. Ma, S. Zhang, and Y . Feng, “LLaMA-Omni: Seamless speech interaction with large language models,”arXiv preprint arXiv:2409.06666, 2024

    2024 arXiv

  2. [10]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D ´efossez, L. Mazar´e, M. Orsini, A. Royer, P. P´erez, H. J´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,”arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  3. [11]

    Dynamic- SUPERB: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech,

    C.-y. Huang, K.-H. Lu, S.-H. Wang, C.-Y . Hsiao, C.-Y . Kuan, H. Wu, S. Arora, K.-W. Chang, J. Shi, Y . Penget al., “Dynamic- SUPERB: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech,” inICASSP 2024-2024 IEEE International Conference ...

    2024

  4. [12]

    Dynamic- SUPERB Phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,

    C.-y. Huang, W.-C. Chen, S.-w. Yang, A. T. Liu, C.-A. Li, Y .-X. Lin, W.-C. Tseng, A. Diwan, Y .-J. Shih, J. Shiet al., “Dynamic- SUPERB Phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,”arXiv preprint arXiv...

    2024 arXiv

  5. [13]

    AudioBench: A universal benchmark for audio large language models,

    B. Wang, X. Zou, G. Lin, S. Sun, Z. Liu, W. Zhang, Z. Liu, A. Aw, and N. F. Chen, “AudioBench: A universal benchmark for audio large language models,”arXiv preprint arXiv:2406.16020, 2024

    2024 arXiv

  6. [14]

    AIR-Bench: Benchmarking large audio- language models via generative comprehension,

    Q. Yang, J. Xu, W. Liu, Y . Chu, Z. Jiang, X. Zhou, Y . Leng, Y . Lv, Z. Zhao, C. Zhouet al., “AIR-Bench: Benchmarking large audio- language models via generative comprehension,”arXiv preprint arXiv:2402.07729, 2024

    2024 arXiv

  7. [15]

    V oiceBench: Benchmarking llm-based voice assistants,

    Y . Chen, X. Yue, C. Zhang, X. Gao, R. T. Tan, and H. Li, “V oiceBench: Benchmarking llm-based voice assistants,”arXiv preprint arXiv:2410.17196, 2024

    2024 arXiv

  8. [16]

    SALMONN:Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “SALMONN:Towards generic hearing abilities for large language models,” inThe Twelfth International Conference on Learning Representations,ICLR 2024, Vienna, Austria, May 7- 11, 2024

    2024

  9. [17]

    SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities,

    D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y . Zhou, and X. Qiu, “SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities,” inFindings of the Associ- ation for Computational Linguistics: EMNLP 2023, 2023, pp. 15 757–15 773

    2023

  10. [18]

    Qwen2-audio technical report,

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Linet al., “Qwen2-audio technical report,”arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  11. [19]

    SNAC: Multi- scale neural audio codec,

    H. Siuzdak, F. Gr ¨otschla, and L. A. Lanzend¨orfer, “SNAC: Multi- scale neural audio codec,” inAudio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation

    2024

  12. [20]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  13. [21]

    HiFi-GAN: Generative adversar- ial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversar- ial networks for efficient and high fidelity speech synthesis,”Ad- vances in neural information processing systems, vol. 33, pp. 17 022–17 033, 2020

    2020

  14. [22]

    Speech resynthesis from discrete disentangled self-supervised representations,

    A. Polyak, Y . Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux, “Speech resynthesis from discrete disentangled self-supervised representations,” inINTER- SPEECH 2021-Annual Conference of the International Speech Communication Association, 2021

    2021

  15. [23]

    Westlake-Omni,

    Xinchen-AI, “Westlake-Omni,” https://github.com/xinchen-ai/ Westlake-Omni, 2024

    2024

  16. [24]

    Freeze- Omni: A smart and low latency speech-to-speech dialogue model with frozen llm,

    X. Wang, Y . Li, C. Fu, L. Xie, K. Li, X. Sun, and L. Ma, “Freeze- Omni: A smart and low latency speech-to-speech dialogue model with frozen llm,”arXiv preprint arXiv:2411.00774, 2024

    2024 arXiv

  17. [25]

    Be- yond turn-based interfaces: Synchronous llms as full-duplex dia- logue agents,

    B. Veluri, B. Peloquin, B. Yu, H. Gong, and S. Gollakota, “Be- yond turn-based interfaces: Synchronous llms as full-duplex dia- logue agents,” inProceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, 2024, pp. 21 390– 21 402

    2024

  18. [26]

    OmniFlatten: An end-to-end GPT model for seamless voice conversation,

    Q. Zhang, L. Cheng, C. Deng, Q. Chen, W. Wang, S. Zheng, J. Liu, H. Yu, and C. Tan, “OmniFlatten: An end-to-end GPT model for seamless voice conversation,”arXiv preprint arXiv:2410.17799, 2024

    2024 arXiv

  19. [27]

    Baichuan-Omni-1.5 technical report,

    Y . Li, J. Liu, T. Zhang, S. Chen, T. Li, Z. Li, L. Liu, L. Ming, G. Dong, D. Panet al., “Baichuan-Omni-1.5 technical report,” arXiv preprint arXiv:2501.15368, 2025

    2025

  20. [28]

    TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension,

    M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer, “TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension,” inProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017, pp. 1601–1611

    2017

  21. [29]

    Lib- rispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  22. [30]

    LibriSQA: A novel dataset and framework for spoken question answering with large language models,

    Z. Zhao, Y . Jiang, H. Liu, Y . Wang, and Y . Wang, “LibriSQA: A novel dataset and framework for spoken question answering with large language models,”IEEE Transactions on Artificial Intelli- gence, 2024

    2024

  23. [31]

    Spoken SQuAD: A study of mitigating the impact of speech recognition errors on listening comprehension,

    C.-H. Lee, S.-L. Wu, C.-L. Liu, and H. yi Lee, “Spoken SQuAD: A study of mitigating the impact of speech recognition errors on listening comprehension,” inInterspeech 2018, 2018, pp. 3459– 3463

    2018

  24. [32]

    IEMOCAP: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, pp. 335–359, 2008

    2008

  25. [33]

    Common V oice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Hen- retty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common V oice: A massively-multilingual speech corpus,” inProceedings of the Twelfth Language Resources and Evaluation Conference, 2020, pp. 4218–4222

    2020

  26. [34]

    Stanford alpaca: an instruction- following llama model (2023),

    R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: an instruction- following llama model (2023),”URL https://github. com/tatsu- lab/stanford alpaca, vol. 1, no. 9, 2023

    2023

  27. [35]

    The t05 sys- tem for the voicemos challenge 2024: Transfer learning from deep image classifier to naturalness mos prediction of high-quality syn- thetic speech,

    K. Baba, W. Nakata, Y . Saito, and H. Saruwatari, “The t05 sys- tem for the voicemos challenge 2024: Transfer learning from deep image classifier to naturalness mos prediction of high-quality syn- thetic speech,”arXiv preprint arXiv:2409.09305, 2024

    2024 arXiv

  28. [36]

    Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

    Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y . Yang, H. Hu, S. Zheng, Y . Gu, Z. Maet al., “Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,”arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  29. [37]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    2023

  30. [38]

    Mini-Omni2: Towards open-source GPT- 4o with vision, speech and duplex capabilities,

    Z. Xie and C. Wu, “Mini-Omni2: Towards open-source GPT- 4o with vision, speech and duplex capabilities,”arXiv preprint arXiv:2410.11190, 2024

    2024 arXiv

  31. [39]

    GLM-4-V oice: Towards intelligent and human-like end- to-end spoken chatbot,

    A. Zeng, Z. Du, M. Liu, K. Wang, S. Jiang, L. Zhao, Y . Dong, and J. Tang, “GLM-4-V oice: Towards intelligent and human-like end- to-end spoken chatbot,”arXiv preprint arXiv:2412.02612, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.