Pith. sign in

REVIEW 5 cited by

VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.04962 v4 pith:HVZBCHBG submitted 2025-01-09 cs.CL cs.SDeess.AS

Wenqian Cui , Xiaoqi Jiao , Ziqiao Meng , Irwin King This is my paper
classification cs.CLcs.SDeess.AS
keywords slmsvoxevalknowledgemodelsaudiobenchmarkconditionsend-to-end
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the rising need for speech-based interaction models, end-to-end Spoken Language Models (SLMs) have emerged as a promising solution. While these models require comprehensive world knowledge for meaningful and reliable human interactions, existing question-answering (QA) benchmarks fall short in evaluating SLMs' knowledge understanding due to their inability to support end-to-end speech evaluation and account for varied input audio conditions. To address these limitations, we present VoxEval, a novel SpeechQA benchmark that assesses SLMs' knowledge understanding through pure speech interactions. Our benchmark 1) uniquely maintains speech format for both inputs and outputs, 2) evaluates model robustness across diverse input audio conditions, and 3) pioneers the assessment of complex tasks like mathematical reasoning in spoken format. Systematic evaluation demonstrates that VoxEval presents significant challenges to current SLMs, revealing their sensitivity to varying audio conditions and highlighting the need to enhance reasoning capabilities in future development. We hope this benchmark could guide the advancement of more sophisticated and reliable SLMs. VoxEval dataset is available at: https://github.com/dreamtheater123/VoxEval

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A new spoken math benchmark, Spoken-MQA, shows that current speech-based AI models reason poorly from spoken math input, especially for arithmetic and knowledge-heavy problems.

  2. Audio-Aware Large Language Models as Judges for Speaking Styles

    eess.AS 2025-06 conditional novelty 6.0 of 10

    Gemini-2.5-Pro can act as an automatic judge of speaking style in speech, with human agreement comparable to human-human agreement.

  3. Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Language models fine-tuned to predict speech reductions and prosodic prominences from text perform above random baselines, and models pretrained on conversational data outperform those pretrained on written data in En...

  4. A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.

  5. BoSS: Beyond-Semantic Speech

    cs.SD 2025-07 conditional novelty 4.0 of 10

    Current spoken-language models perform poorly on a new five-task evaluation of beyond-semantic speech signals, including dialect, emotion, age, and non-verbal cues.

Pith tools