Pith. sign in

REVIEW 3 major objections 7 minor 6 cited by

A Preliminary Exploration with GPT-4o Voice Mode

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This report evaluates GPT-4o's audio understanding across 180 tasks and three benchmarks, arguing that it leads current audio-language models in speech and music reasoning and hallucination resistance, while its built-in safety refusals…

desk verdict First broad public capability map of GPT-4o voice mode, with a genuinely interesting refusal analysis, but the comparative scores hinge on an unstated refusal-scoring rule and an undisclosed benchmark overlap. read the letter →

arxiv 2502.09940 v1 pith:NR56754H submitted 2025-02-14 cs.CL cs.SDeess.AS

Yu-Xiang Lin , Chih-Kai Yang , Wei-Chih Chen , Chen-An Li , Chien-yu Huang , Xuanjun Chen , Hung-yi Lee This is my paper · ORCID
classification cs.CLcs.SDeess.AS
keywords GPT-4oaudio-languagemodelsspeechunderstandingaudiohallucinationmultimodalbenchmarkDynamic-SUPERBMMAUCMM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to answer how close end-to-end large audio-language models are to becoming universal instruction-following speech models, using GPT-4o as the test case. It evaluates the model on three public benchmarks: Dynamic-SUPERB Phase 2's 180 tasks, MMAU's expert-written audio reasoning questions, and CMM's hallucination tests. The authors claim that GPT-4o shows strong audio, speech, and music understanding, achieving the best average score on MMAU among the compared models and unusually high resistance to audio hallucinations. They also find that GPT-4o fails on tasks such as audio duration prediction and instrument classification, and that its safety post-training makes it refuse many requests that do not obviously raise safety concerns. The report concludes that model performance varies with evaluation protocol, and that refusal behavior must be reported alongside accuracy to interpret benchmark scores honestly.

What carries the argument

The argument is carried by three measurement instruments and one refusal detector. Dynamic-SUPERB Phase 2 contributes 180 community-built tasks spanning speech, audio, and music, with relative scores computed against a Whisper-LLaMA cascade baseline and, for classification tasks, against a repeated random-guess baseline. MMAU contributes expert-written multiple-choice questions across audio, music, and speech domains, giving a direct comparison of reasoning and knowledge. CMM contributes the perception accuracy and hallucination resistance metrics that support the central hallucination claim. The refusal detector, combining template string matching with an LLM judge, separates cases where GPT-4o will not answer from cases where it answers incorrectly, which is essential because many of the model's lowest scores are refusals rather than wrong predictions.

What would settle it

Re-run every Dynamic-SUPERB and MMAU item multiple times with varied prompting and sampling, then measure the variance in accuracy and refusal rate; if scores swing widely across runs or reworded instructions, then the claimed strengths and weaknesses are properties of the protocol rather than of the models.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a proprietary end-to-end audio-language model can combine speech, audio, and music understanding with strong instruction following and markedly lower hallucination rates than open large audio-language models, but that its built-in safeguards systematically refuse a class of tasks, including speaker identification, age classification, MOS prediction, and audio deepfake detection. The supporting numbers are an MMAU Test average of 60.46% versus 52.97% for the next proprietary baseline and 52.50% for the best open model, and a CMM hallucination resistance of 83.75% versus 59% for the next-best measured model. On Dynamic-SUPERB, GPT-4o leads on intent classification, stress detection, stuttering detection, multilingual speech recognition, and singing analysis, yet posts negative relative scores on phonological feature classification, audio duration prediction, and several music tasks. The paper treats these scores as a preliminary map of what current LALMs can and cannot do, not as a final verdict on GPT-4o.

Load-bearing premise

The whole comparison stands on the assumption that running the official benchmark pipelines once per sample produces scores that are stable and representative of each model's true ability.

Editorial extensions

If this is right

  • If the MMAU numbers hold, GPT-4o voice mode is the strongest audio-language model in this comparison, ahead of the other proprietary model and all open baselines.
  • If the CMM numbers hold, GPT-4o's hallucination resistance is substantially higher than every other measured model, even though its perception accuracy is lower than the best open model.
  • GPT-4o's refusal pattern means current benchmark leaderboards for proprietary LALMs conflate capability with safety willingness, so future comparisons should report refusal rates alongside accuracy.
  • Several Dynamic-SUPERB tasks remain at or below random-guess accuracy for all LALMs, showing that universal instruction-following speech models are still far from solved.
  • Tasks that depend mostly on text-transcribable content can still be solved well by cascaded systems, so end-to-end LALMs earn their advantage mainly on acoustic, prosodic, and musical information.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The near-total refusals on tasks where even strong models perform near chance suggest refusal rate may function as a model-internal signal of low confidence, meaning refusal patterns could be mined as a cheap difficulty label for benchmark items.
  • Editorial inference: The finding that random guessing beats many LALMs on hard classification tasks raises the possibility that multiple-choice formats understate generative audio understanding; free-form responses with rubric-based scoring could show a different capability profile.
  • Editorial inference: The dataset-dependent refusal rates for fundamentally the same task, such as speaker verification, imply that safety post-training is prompt-sensitive; systematically varying instruction wording would map each model's refusal boundary more precisely before comparing models against each other.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This report evaluates GPT-4o voice mode (gpt-4o-audio-preview-2024-10-01) on three benchmarks: Dynamic-SUPERB Phase 2, MMAU, and CMM. For Dynamic-SUPERB, the authors compute relative scores against Whisper-LLaMA and random baselines and report refusal rates for each task domain. The central qualitative claims are that GPT-4o shows strong audio, speech, and music understanding on several task families (intent classification, multilingual ASR, singing analysis, and hallucination resistance on CMM), while it struggles with duration prediction and instrument classification and frequently refuses tasks related to speaker traits, health inference, deepfake detection, and music analysis. The paper repeatedly frames itself as a preliminary exploration and notes sensitivity to evaluation protocol.

Significance. If the core comparisons are sound, this is a useful early snapshot of a proprietary end-to-end audio-language model, with the additional value of quantifying refusal behavior rather than discarding it. The authors should be credited for reporting refusal rates with two detectors, for introducing a random baseline to contextualize relative scores, and for explicitly acknowledging protocol sensitivity. However, the headline claims are only as reliable as the scoring protocol; the current manuscript leaves the treatment of refusals unspecified, does not define the relative-score metric, and provides no uncertainty quantification.

major comments (3)
  1. [§4 intro, Fig. 3, Fig. 25] The relative score is never defined. The text says the computation 'follow[s] the setting in Dynamic-SUPERB' but gives no formula or normalization. This matters because the metric produces extreme values when the Whisper-LLaMA baseline is near floor: LTU-AS receives -649.46 on Phoneme Segment Counting (abs diff) in Fig. 3, and SALMONN-13B receives -1771.85 on Audio Duration Prediction in Fig. 25. Define the metric, report the raw scores, and flag or exclude tasks where the baseline denominator is near zero so that the relative numbers are not read as meaningful effect sizes.
  2. [§3.1, §4–§6] All GPT-4o results appear to come from a single run with no confidence intervals or repeated sampling. Given the observed variability in refusal behavior and API stochasticity, the comparisons—especially small relative-score differences and refusal-rate differences across datasets—are difficult to interpret without uncertainty quantification. The 100-run random baseline does not address model variance; at minimum, the authors should state whether the API calls were repeated and report standard errors or a sensitivity analysis.
  3. [§3.1, ref [8], author list] The authors are also contributors to Dynamic-SUPERB (ref [8] shares multiple co-authors with this paper) and evaluate GPT-4o on that benchmark using the official scripts, but no conflict-of-interest or contribution disclosure is made. Add an explicit statement identifying the authors' role in constructing the benchmark and the specific tasks or scripts to which they contributed, so that readers can assess the risk of benchmark-construction bias.
minor comments (7)
  1. [Table 2, §4.2] The task name 'L2 English Accuracy/Fluency/Prodosy Ranking' misspells 'Prosody'; the same typo appears in the text in §4.2.
  2. [§4 intro] The sentence 'we simply repeats the experiments for 100 times' should read 'we simply repeat the experiments 100 times.'
  3. [§4.5] Figure 9 and Figure 10 are referenced in reverse order in the text ('Table 5, Figure 10 and Figure 9 respectively'); please correct the ordering or the figure numbering.
  4. [§4.5, §4.8] 'Whipser-LLaMA' is a typo for 'Whisper-LLaMA' in both sections.
  5. [§4.6] The phrase 'reflecting the the wealth of textual knowledge' contains a duplicated 'the.'
  6. [Fig. 25] The GPT-4o entry for 'Audio Duration Prediction' appears blank in the relative-score figure; clarify whether this is a formatting artifact, a refusal, or an N/A case.
  7. [§4.1] The claim that Random baseline outperforming LALMs 'supports our conjecture that GPT-4o lacks confidence' is not directly supported by that observation; the refusal-confidence link is speculative and should be flagged as such.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GPT-4o's scores are empirical measurements against benchmarks whose ground truth is external to the evaluated model, so the central claims do not reduce to the paper's own inputs.

full rationale

This paper is an empirical evaluation report rather than a derivation, so there is no derivation chain whose outputs are defined in terms of its inputs. GPT-4o's performance on Dynamic-SUPERB, MMAU, and CMM is obtained by running a fixed proprietary model against benchmark items with independently fixed labels and official scoring scripts. Several authors are also contributors to Dynamic-SUPERB [8], which is a self-citation, but it is not load-bearing in a circular sense: the benchmark's ground truth is external to the evaluated model, and no parameter of GPT-4o is fitted to those labels here. The refusal-rate analysis in Section 3.2 and the relative-score tables (e.g., Section 4.2) are arithmetic or rule-based transformations of collected responses, not predictions derived from fitted quantities. The extreme relative scores for near-zero baselines, such as LTU-AS's -649.46 on phoneme segment counting, indicate metric instability rather than circularity. The paper repeatedly acknowledges that model performance varies with evaluation protocols (Abstract, Section 7) and notes that some tasks lack official protocols (Section 4.1 footnote) or yield 100% N/A rates (Sections 4.11 and 4.13). It does not explicitly state how refusals are scored in Dynamic-SUPERB, but that is a reproducibility and validity concern, not a circularity concern. No equation or definition in the paper makes the claimed conclusion true by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. It relies on three benchmark suites as ground truth, on a relative-score metric that is sensitive to baseline performance, and on two refusal-detection heuristics. The main unstated premise is that these benchmarks and heuristics produce a fair comparison for a proprietary model that frequently refuses tasks.

assumptions (4)
  • domain assumption Dynamic-SUPERB, MMAU, and CMM provide valid ground truth and fair evaluation pipelines for comparing LALMs.
    Sections 4-6 use these benchmark scores as the evidence for all claims about GPT-4o's capabilities.
  • domain assumption Relative score to Whisper-LLaMA is a meaningful metric even when the baseline's performance is near zero.
    Section 4.2 reports extreme relative scores like -649.46 for Phoneme Segment Counting, indicating the metric is unstable for near-zero baselines.
  • domain assumption The refusal detection methods (string matching and LLaMA-3.1-8B-Instruct) correctly identify refusals across tasks.
    Section 3.2 introduces these methods; in several domains (4.2, 4.8, 4.12) the two methods disagree and the authors manually adjudicate some but not all cases.
  • domain assumption Single evaluation runs of GPT-4o are representative, i.e., the model's responses are stable across API calls.
    The paper reports no repeated runs or error bars for GPT-4o on the benchmarks (Sections 4-6).

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Preliminary Exploration with GPT-4o Voice Mode." pith.science (2026). https://pith.science/paper/NR56754H

@misc{pith2026250209940,
  author       = {Pith},
  title        = {Pith review of: A Preliminary Exploration with GPT-4o Voice Mode},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NR56754H}},
  note         = {Machine review of arXiv:2502.09940}
}
read the original abstract

With the rise of multimodal large language models, GPT-4o stands out as a pioneering model, driving us to evaluate its capabilities. This report assesses GPT-4o across various tasks to analyze its audio processing and reasoning abilities. We find that GPT-4o exhibits strong knowledge in audio, speech, and music understanding, performing well in tasks like intent classification, spoken command classification, semantic and grammatical reasoning., multilingual speech recognition, and singing analysis. It also shows greater robustness against hallucinations than other large audio-language models (LALMs). However, it struggles with tasks such as audio duration prediction and instrument classification. Additionally, GPT-4o's safety mechanisms cause it to decline tasks like speaker identification, age classification, MOS prediction, and audio deepfake detection. Notably, the model exhibits a significantly different refusal rate when responding to speaker verification tasks on different datasets. This is likely due to variations in the accompanying instructions or the quality of the input audio, suggesting the sensitivity of its built-in safeguards. Finally, we acknowledge that model performance varies with evaluation protocols. This report only serves as a preliminary exploration of the current state of LALMs.

Figures

Figures reproduced from arXiv: 2502.09940 by the authors.

Figure 1
Figure 1. Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Refusal rates of tasks in the (Speech) Paralinguistics domain 6 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (27 more)
Figure 4
Figure 4. Figure 4: Refusal rates of tasks in the (Speech) Phonetics, Phonology, Prosody domain. The abbreviation “PFC” in the plot refers to “Phonological Feature Classification” 9 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Refusal rates of tasks in the (Speech) Safety & Security domain. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Refusal rates of tasks in the (Speech) Speaker & Language domain. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Refusal rates of tasks in the (Speech) Speech Enhancement domain. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Refusal rates of tasks in the (Speech) Speech Recognition domain. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 15
Figure 15. Figure 15: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Refusal rates of tasks in the (Speech) Spoken Language Understanding domain. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Refusal rates of tasks in the (Music) Harmony & Pitch domain. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: Refusal rates of tasks in the (Music) Music Classification domain. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Refusal rates of tasks in the (Music) Rhythm Analysis domain. 4.12 Audio Domain - Quality Assessment The refusal rates of tasks about Quality Assessment are demonstrated in [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22: Refusal rates of tasks in the (Audio) Quality Assessment domain. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_22.png]
Figure 23
Figure 23. Figure 23: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p027_23.png]
Figure 24
Figure 24. Figure 24: Refusal rates of tasks in the (Audio) Safety domain. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_24.png]
Figure 25
Figure 25. Figure 25: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p028_25.png]
Figure 26
Figure 26. Figure 26: Refusal rates of tasks in the (Audio) Signal-Characteristics Analysis domain. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_26.png]
Figure 27
Figure 27. Figure 27: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p029_27.png]
Figure 28
Figure 28. Figure 28: Refusal rates of tasks in the [PITH_FULL_IMAGE:figures/full_fig_p029_28.png]
Figure 29
Figure 29. Figure 29: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p031_29.png]
Figure 30
Figure 30. Figure 30: Refusal rates of tasks in the (Audio) Sound Event domain. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_30.png]
Figure 31
Figure 31. Figure 31: Relative performance comparison of models in the [PITH_FULL_IMAGE:figures/full_fig_p032_31.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Models

    cs.SD 2026-07 conditional novelty 7.0 of 10

    Label-free real-vs-noise scoring of audio-encoder neurons, followed by sparse amplification, substantially improves LALM perception of non-semantic speech attributes without retraining.

  2. Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Lychee-FD resolves modality interference in full-duplex spoken language models by separating acoustic and semantic parameters in deep layers and adding a dense semantic alignment channel, achieving state-of-the-art pe...

  3. AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    By projecting hidden states to the vocabulary at every layer, the paper shows that failed attribute recognition in three LALMs is marked by mid-network information peaks followed by degradation, and that models rely o...

  4. Towards Generalized Source Tracing for Codec-Based Deepfake Speech

    cs.SD 2025-06 conditional novelty 5.0 of 10

    SASTNet, which fuses Whisper semantic features with Wav2Vec2 and AudioMAE acoustic features, improves source tracing for codec-based deepfake speech on CodecFake+, while exposing that prior models overfit to silence.

  5. A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.

  6. Breaking the Barriers of Text-Hungry and Audio-Deficient AI

    cs.SD 2025-06 reject novelty 4.0 of 10

    A proposed audio-native translation framework called MAST with fractional diffusion is described, but no evidence is given that it produces working translations.

Reference graph

Works this paper leans on

31 extracted references · 10 canonical work pages · cited by 6 Pith papers

  1. [8]

    Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks

    Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang, Andy T Liu, Chen-An Li, Yu-Xiang Lin, Wei-Cheng Tseng, Anuj Diwan, Yi-Jen Shih, Jiatong Shi, et al. Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks. arXiv preprint arXiv:2411.05361, 2024

    arXiv 2024

  2. [1]

    Qwen2-audio technical report

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuan- jun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024

    arXiv 2024

  3. [2]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023

    arXiv 2023

  4. [3]

    Moshi: a speech-text foundation model for real-time dialogue

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037, 2024. 34

    arXiv

  5. [4]

    Audio entailment: Assessing deductive reasoning for audio understanding

    Soham Deshmukh, Shuo Han, Hazim Bukhari, Benjamin Elizalde, Hannes Gamper, Rita Singh, and Bhiksha Raj. Audio entailment: Assessing deductive reasoning for audio understanding. arXiv preprint arXiv:2407.18062, 2024

    arXiv 2024

  6. [5]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  7. [6]

    Llama-omni: Seamless speech interaction with large language models

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06666, 2024

    arXiv 2024

  8. [7]

    Wavllm: Towards robust and adaptive speech large language model

    Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, et al. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint arXiv:2404.00656, 2024

    arXiv 2024

Show all 31 references
  1. [9]

    Dynamic-superb: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech

    Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi-Yuan Hsiao, Chun-Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, et al. Dynamic-superb: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech. In ICASSP 2024-2024 I...

    2024

  2. [10]

    Audiogpt: Understanding and generating speech, music, sound, and talking head

    Rongjie Huang et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 23802–23804, 2024

    2024

  3. [11]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  4. [12]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022–17033, 2020

    2020

  5. [13]

    Speech-copilot: Leveraging large language models for speech processing via task decomposition, modularization, and program generation

    Chun-Yi Kuan, Chih-Kai Yang, Wei-Ping Huang, Ke-Han Lu, and Hung-yi Lee. Speech-copilot: Leveraging large language models for speech processing via task decomposition, modularization, and program generation. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 1060–10...

    2024

  6. [14]

    Rlaif vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif vs. rlhf: scaling reinforcement learning from human feedback with ai feedback. In Proceedings of the 41...

    2024

  7. [15]

    The curse of multi-modalities: Evaluating hallucinations of large multimodal models across language, visual, and audio

    Sicong Leng, Yun Xing, Zesen Cheng, Yang Zhou, Hang Zhang, Xin Li, Deli Zhao, Shijian Lu, Chunyan Miao, and Lidong Bing. The curse of multi-modalities: Evaluating hallucinations of large multimodal models across language, visual, and audio. arXiv preprint arXiv:2410.12787, 2024

    arXiv

  8. [16]

    Align-slm: Textless spoken language models with reinforcement learning from ai feedback

    Guan-Ting Lin, Prashanth Gurunath Shivakumar, Aditya Gourav, Yile Gu, Ankur Gandhe, Hung- yi Lee, and Ivan Bulyko. Align-slm: Textless spoken language models with reinforcement learning from ai feedback. arXiv preprint arXiv:2411.01834, 2024

    2024 arXiv

  9. [17]

    Music understand- ing llama: Advancing text-to-music generation with question answering and captioning

    Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. Music understand- ing llama: Advancing text-to-music generation with question answering and captioning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages...

    2024

  10. [18]

    Generative spoken dialogue language modeling

    Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11:250–266, 2023

    2023

  11. [19]

    Gpt-4o system card

    OpenAI. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  12. [20]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  13. [21]

    Mmau: A massive multi-task audio understanding and reasoning benchmark

    S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410.19168, 2024

    arXiv

  14. [22]

    Salmonn: Towards generic hearing abilities for large language models

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, MA Zejun, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  15. [23]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  16. [24]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  17. [25]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  18. [26]

    Hear: Holistic evaluation of audio representations

    Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W Schuller, Christian J Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, et al. Hear: Holistic evaluation of audio representations. In NeurIPS 2021 Competitions and Demonstrations Track, ...

    2021

  19. [27]

    Lin, Andy T

    Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y . Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman M...

    2021

  20. [28]

    Mini-omni: Language models can hear, talk while thinking in streaming

    Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408.16725, 2024

    2024 arXiv

  21. [29]

    Marble: Music audio representation benchmark for universal evaluation

    Ruibin Yuan, Yinghao Ma, Yizhi Li, Ge Zhang, Xingran Chen, Hanzhi Yin, Yiqi Liu, Jiawen Huang, Zeyue Tian, Binyue Deng, et al. Marble: Music audio representation benchmark for universal evaluation. Advances in Neural Information Processing Systems, 36:39626–39647, 2023

    2023

  22. [30]

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000, 2023

    2023 arXiv

  23. [31]

    Speechalign: Aligning speech generation to human preferences

    Dong Zhang, Zhaowei Li, Shimin Li, Xin Zhang, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechalign: Aligning speech generation to human preferences. arXiv preprint arXiv:2404.05600, 2024. 36

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.