Pith. sign in

REVIEW 4 major objections 5 minor 11 cited by

AHELM: A Holistic Evaluation of Audio-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AHELM, a ten-aspect standardized benchmark for audio-language models, reports that no single model leads everywhere: Gemini 2.5 Pro tops five aspects but shows a statistically significant female-vs-male transcription gap, and a simple speec

desk verdict A genuinely useful and transparent benchmark for audio-language models, with a solid core and a few overstated findings—safety especially—that need revision before the ranking claims are taken at face value. read the letter →

arxiv 2508.21376 v2 pith:TIBXHO4Y submitted 2025-08-29 cs.AI cs.CL

Tony Lee , Haoqin Tu , Chi Heem Wong , Zijun Wang , Siwei Yang , Yifan Mai , Yuyin Zhou , Cihang Xie
show 1 more author
Percy Liang
This is my paper
classification cs.AIcs.CL
keywords audio-languagemodelsbenchmarkholisticevaluationfairnessbiasreasoningrobustnesssafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that audio-language models (ALMs) can be evaluated fairly and comprehensively in one standardized benchmark, and that doing so reveals a fragmented capability landscape rather than a single best model. It assembles 14 existing datasets and two new synthetic ones into AHELM, covering perception, knowledge, reasoning, emotion, bias, fairness, multilinguality, robustness, toxicity, and safety. Fourteen ALMs and three ASR+LM baselines are run under identical prompts, temperature, and metrics. The results show Gemini 2.5 Pro ranked first in five aspects and best overall, yet it and a few other models transcribe female speech significantly worse than male speech; meanwhile one baseline that only transcribes speech then calls a language model ranks sixth overall. The authors argue this makes AHELM useful for model developers and deployers and for exposing where ALMs still lose to simpler pipelines.

What carries the argument

The operating machinery is AHELM's aspect-scenario-adaptation-metric loop. Each dataset is registered as one or more scenarios; every model gets the same zero-shot adaptation; metrics are chosen per scenario; and mean win rate aggregates head-to-head comparisons. Two new datasets carry specific load: PARADE, multiple-choice audio with occupation/status pairs and male/female voices, operationalizes bias by treating 'unclear' as the unbiased answer; CoRe-Bench, text-to-speech-rendered multi-speaker dialogues with inference questions plus unanswerable distractors, operationalizes conversational reasoning and hallucination. The ASR+LM baseline chain (Whisper-1 or GPT-4o Transcribe into GPT-4o) a

What would settle it

Take a larger random sample of AudioCaps and AIR-Bench Chat outputs (1,000+ instances), have several human raters score them with the same 1-5 rubric, and compare rankings produced by the GPT-4o judge versus human scores; if exact agreement stays near 50% or a different judge changes model ordering, the paper's open-ended leaderboard positions are not settled. Separately, re-run the FLEURS fairness paired t-test on a fresh or full test split to see whether Gemini 2.5 Pro's female-speech advantage (p=0.02) replicates.

Watch

Extended reading notes

Core claim

AHELM's central claim is that holistic evaluation is possible with a fixed protocol: map each benchmark dataset to one of ten aspects, prompt every model zero-shot with the same text, decode at temperature 0, score with exact match, WER/BLEU, or a GPT-4o judge, and aggregate into mean win rates. Under that protocol no model wins everything; Gemini 2.5 Pro leads the aggregate leaderboard but is not top in five aspects, and the fairness analysis flags a real group disparity on ASR (paired p=0.02 on FLEURS in the main text, with the abstract citing p=0.01). The benchmark also introduces PARADE, synthetic audio with gender-controlled speech asking whether a speaker is more likely a programmer or

Load-bearing premise

The load-bearing premise is that GPT-4o's automated judgments of open-ended audio answers are trustworthy enough to rank models; the paper checked this on only 197 instances, with 50.8% exact agreement and 83.8% within-one-point agreement with human raters.

Editorial extensions

If this is right

  • Standardized prompts, temperature, and metrics let developers compare models within and across families on equal footing, including across Gemini 1.5→2.5 and GPT-4o Audio versions.
  • Simple ASR+LM pipelines are credible alternatives for speech-centric ALM tasks, ranking 2nd to 6th in several aspects; ALMs must justify their extra complexity on non-speech or prosody-heavy tasks.
  • Gemini 2.5 Pro's ASR performance is not group-fair by sex (p≈0.01–0.02 depending on test), so an overall winner can still require fairness auditing before deployment.
  • Open-weight ALMs such as Qwen models lag in instruction following, which depresses their scores independently of audio understanding.
  • OpenAI audio models refuse voice jailbreak attempts at 96.7–99.4% while Gemini and Qwen models refuse only 25–53%, indicating safety behavior is not correlated with overall capability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The automated judge's 50.8% exact agreement with humans means small differences between models on open-ended aspects may not be meaningful; a practical next step is publishing per-instance judge scores and rerunning the leaderboard with alternate judges.
  • The ASR+LM baselines' strong results suggest a design direction: hybrid ALMs that embed dedicated ASR modules could close the robustness gap without giving up end-to-end audio reasoning.
  • Because PARADE and CoRe-Bench are generated synthetically, the same pipeline can be extended to other protected attributes (accent, age, dialect) and other reasoning types, making bias and reasoning evaluation scalable.
  • MuTox language differences (e.g., worst scores on English and Vietnamese) could reflect dataset difficulty rather than model toxicity skill; a calibrated per-language analysis would separate language-specific curation from genuine capability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. AHELM is a benchmark for audio-language models (ALMs) that aggregates 14 existing datasets plus two new synthetic datasets (PARADE for bias, CoRe-Bench for conversational reasoning) to evaluate 10 aspects: audio perception, knowledge, reasoning, emotion detection, bias, fairness, multilinguality, robustness, toxicity, and safety. The authors standardize prompts, inference parameters, and metrics, and evaluate 14 ALMs plus 3 ASR+LM baselines, releasing all prompts and outputs. The central claims are that no single model dominates; Gemini 2.5 Pro ranks first overall (mean win rate 0.803) but shows gender-based ASR unfairness (abstract reports p=0.01, intro p=0.02); simple ASR+LM baselines are competitive (one ranks 6th overall); and open-weight models lag in instruction following.

Significance. If the identified validity concerns are addressed, AHELM would be a valuable community resource: it extends the HELM framework to audio, standardizes evaluation across many aspects, provides transparent release of raw prompts and generations, introduces two new datasets, and includes simple baselines that give insight into what ALMs add over ASR+LM pipelines. These are concrete strengths that make the benchmark potentially useful for model developers and deployers. The paper's contribution is primarily empirical and organizational rather than theoretical; its impact depends on the reliability of the automated judge and the fairness/safety analyses, which currently have load-bearing weaknesses.

major comments (4)
  1. [Table 2 / Table A23 / Result 9] The Safety aspect uses the 'Voice jailbreak attacks' scenario from [38], a dataset constructed specifically to attack GPT-4o. Reporting refusal rates across all models (Table A23) and concluding 'OpenAI's models are better at defending against jailbreak attacks' (Result 9) is confounded by attack transferability: the prompts are calibrated to GPT-4o's audio stack and may have been patched in the evaluated OpenAI versions, as the paper itself concedes in Table A23. Since Safety is one of the 10 aspects defining 'holistic,' this undermines the cross-model fairness claim for that aspect. Please either rename the scenario to 'resistance to the Voice Jailbreak Attacks set [38]' and restrict claims accordingly, or add additional safety scenarios not tied to a single provider.
  2. [§3.2 and Appendix F.3] GPT-4o-as-judge is used for AudioCaps and Air-Bench Chat (knowledge and reasoning). Validation on 197 instances gives exact agreement 50.8% with human raters, ±1 agreement 83.8%, and weighted κ reported as 83.3% in §3.2 but 83.8% in Table A7. Exact agreement of 50.8% means the judge disagrees with humans on roughly half of the instances; this is weak support for using 1–5 scores to rank models, especially when leaderboard differences are small. Please report score distributions and per-category agreement, and provide a sensitivity analysis showing whether rankings change when ±1 is treated as agreement or when a different judge (e.g., Claude) is used. Also fix the κ inconsistency.
  3. [Abstract, §5 Result 6, Table A13] The abstract states that Gemini 2.5 Pro exhibits group unfairness with p=0.01 on ASR tasks, but Table A13 reports p=0.02 for the paired t-test on FLEURS and p=0.34 for the independent t-test. Furthermore, the paper runs paired and independent t-tests for 17 models (34+ tests) without multiple-comparison correction; at α=0.05 one expects roughly 1.7 false positives by chance. The conclusion that 'most other models do not' exhibit unfairness is therefore not statistically supported. Please report multiplicity-adjusted p-values (e.g., FDR or Holm-Bonferroni) and align the abstract, introduction, and tables on the same numbers.
  4. [Table 2, §3.1, Appendix E] The PARADE dataset is described as containing 436 instances in Table 2, 938 examples in §3.1, and Appendix E statistics that sum to 938 (738 occupational + 200 status). This inconsistency directly affects the credibility of the bias aspect. Please provide the exact instance count used in the evaluation, including any subsampling, since §4 states up to 1,000 instances per scenario are evaluated.
minor comments (5)
  1. [Table 2 / §3.2] The metric 'Pseudo-exact match' for CoRe-Bench is not defined in §3.2 or elsewhere. Please define how it differs from exact match and how it is computed.
  2. [Appendix B, Appendix H.6, Table A7] Typos and small inconsistencies: 'Samping Rates' in Appendix B should be 'Sampling Rates'; 'LibreSpeech' in Table A14 and Result 6 should be 'LibriSpeech'; 'Cohan's Kappa' in Table A7 should be 'Cohen's Kappa'; the checklist mentions 'Appendix E.6' for fairness where the material is in Appendix H.6.
  3. [Figures A19/A20 and Table A1] Figures A19 and A20 list Gemini 2.5 Pro (03-25 preview) and Gemini 2.0 Pro (02-05 preview), and Table A1 also lists them, but these models are not in Table 3 nor in the detailed result tables. Please either remove them from the figures/table or clarify if they were evaluated and where their results are.
  4. [§5 Result 6 and Table A14] The text calls p=0.06 'statistically significant' in the phrase 'p=0.06 for Gemini 2.0 Flash' context, while the table marks p<0.1 with an asterisk. The significance threshold is not stated; using 0.05 as the standard would make p=0.06 non-significant. Please state the chosen α and apply it consistently.
  5. [Tables A20–A22] Several cells report 'nan' (e.g., Gemini 2.5 Flash on Finnish in Table A21). The presence of NaN is not explained; if a model failed to produce parseable output, that should be documented and handled consistently in the aggregation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AHELM is a measurement/aggregation benchmark; its rankings are empirical summaries of external datasets, not predictions derived from fitted inputs or self-citations.

full rationale

AHELM does not claim to derive model rankings from first principles; it measures them on fixed, externally sourced datasets (AudioCaps, LibriSpeech, FLEURS, MuTox, etc.) with standardized prompts and metrics. The two new datasets are synthetic but produced by explicit pipelines with cross-model validation (CoRe-Bench uses one LLM to generate and another to validate answerability; PARADE transcripts are human-filtered) and their ground truths are not defined in terms of the target models' outputs. The GPT-4o judge for open-ended tasks is validated against 197 human ratings (exact 50.8%, ±1 83.8%, weighted κ 83.3%), an independent grounding; the paper also discloses unresolved judge-stability concerns ('we have yet to explore how the use of different judges will impact the stability of the leaderboards', App. F.3), which is a limitation, not a circular step. The safety scenario is taken from 'Voice Jailbreak Attacks Against GPT-4o' [38], so the ranking on that single scenario may reflect attack transferability/patching; the authors themselves note 'It may be possible that this vulnerability has specifically been patched by OpenAI since the original paper [38] demonstrated successful attacks against GPT-4o' (App. H.10). This is a construct-validity concern, not a definitional reduction: the attack prompts are external inputs and the refusal rates are empirical outcomes. Citations to HELM/VHELM are framework lineage and are not load-bearing premises that force the results. Consequently, no circular step meeting the paper's evidentiary bar is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 2 invented entities

The central claim rests on the choice of aspects, the validity of the automated judge, and the adequacy of synthetic audio. No numeric free parameters are fitted to data; the benchmark uses aggregate win rates and standard significance tests.

assumptions (3)
  • domain assumption The 10 identified aspects (audio perception, knowledge, reasoning, emotion detection, bias, fairness, multilinguality, robustness, toxicity, safety) are the important dimensions for ALM evaluation.
    Section 3.1. The authors choose these aspects; other dimensions may be missed, acknowledged in Section 6.1.
  • domain assumption GPT-4o as an LM judge provides valid scores for open-ended audio scenarios.
    Appendix F. Human validation shows only 50.8% exact agreement and 83.8% +-1 agreement. The judge may have biases.
  • domain assumption Synthetic text-to-speech audio in PARADE and CoRe-Bench is adequate to measure real-world bias and reasoning.
    Appendices D and E. Conversations are generated by LLMs and voiced by TTS, so they may lack natural prosody and variability.
invented entities (2)
  • PARADE independent evidence
    purpose: Synthetic audio-text dataset to measure stereotype bias in ALMs.
    Released on HuggingFace; usable for external evaluation.
  • CoRe-Bench independent evidence
    purpose: Synthetic multi-turn conversational audio dataset to measure reasoning.
    Released on HuggingFace; usable for external evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AHELM: A Holistic Evaluation of Audio-Language Models." pith.science (2026). https://pith.science/paper/TIBXHO4Y

@misc{pith2026250821376,
  author       = {Pith},
  title        = {Pith review of: AHELM: A Holistic Evaluation of Audio-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TIBXHO4Y}},
  note         = {Machine review of arXiv:2508.21376}
}
abstract

Evaluations of audio-language models (ALMs) -- multimodal models that take interleaved audio and text as input and output text -- are hindered by the lack of standardized benchmarks; most benchmarks measure only one or two capabilities and omit evaluative aspects such as fairness or safety. Furthermore, comparison across models is difficult as separate evaluations test a limited number of models and use different prompting methods and inference parameters. To address these shortfalls, we introduce AHELM, a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE, which evaluates the ALMs on avoiding stereotypes, and CoRe-Bench, which measures reasoning over conversational audio through inferential multi-turn question answering -- to holistically measure the performance of ALMs across 10 aspects we have identified as important to the development and usage of ALMs: audio perception, knowledge, reasoning, emotion detection, bias, fairness, multilinguality, robustness, toxicity, and safety. We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models. We test 14 open-weight and closed-API ALMs from 3 developers and 3 additional simple baseline systems each consisting of an automatic speech recognizer and a language model. Our results show that while Gemini 2.5 Pro ranks top in 5 out of 10 aspects, it exhibits group unfairness ($p=0.01$) on ASR tasks whereas most of the other models do not. We also find that the baseline systems perform reasonably well on AHELM, with one ranking 6th overall despite having only speech-to-text capabilities. For transparency, all raw prompts, model generations, and outputs are available on our website at https://crfm.stanford.edu/helm/audio/v1.0.0. AHELM is intended to be a living benchmark and new datasets and models will be added over time.

Figures

Figures reproduced from arXiv: 2508.21376 by the authors.

Figure 1
Figure 1. AHELM is a comprehensive benchmark that evaluates audio-language models (ALMs) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Evaluation components. Each evaluation run consists of an aspect (i.e., an evaluative dimension), a scenario (i.e., backed by a specific dataset), a model with an adaptation process (i.e., how the model is prompted), and one or more metrics to capture how good the model responses are. An aspect refers to a particular evaluative dimension that aids in assessing overall performance. In AHELM, the aspects considered in… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. VoxSafeBench: Not Just What Is Said, but Who, How, and Where

    cs.SD 2026-04 unverdicted novelty 8.0 of 10

    VoxSafeBench reveals that speech language models recognize social norms from text but fail to apply them when acoustic cues like speaker or scene determine the appropriate response.

  2. RedVox: Safety and Fairness Gaps in Speech Models Across Languages

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    RedVox benchmark shows speech model safety and fairness vulnerabilities persist under non-adversarial conditions, worsen in non-English languages, and increase with spoken inputs.

  3. Toward Fair Speech Technologies: A Comprehensive Survey of Bias and Fairness in Speech AI

    eess.AS 2026-05 accept novelty 7.0 of 10

    The paper delivers a unified framework for fairness in speech technologies by formalizing seven definitions, organizing research into three paradigms, diagnosing pipeline-specific biases, and mapping mitigations to th...

  4. PRiSM: Benchmarking Phone Realization in Speech Models

    cs.CL 2026-01 conditional novelty 7.0 of 10

    PRiSM benchmarks phone recognition in speech models with intrinsic transcription and extrinsic downstream probes, finding that multilingual training and encoder-CTC architectures perform most consistently while LALMs ...

  5. VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech

    eess.AS 2026-04 conditional novelty 6.5 of 10

    Open-ended evaluation with real speech reveals that gender and accent cues cause statistically significant, task-dependent distributional shifts in recommendations from 12 large audio-language models.

  6. RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems

    cs.SD 2026-07 conditional novelty 6.0 of 10

    No voice AI system dominates all capabilities; naturalness, expressiveness, identity stability, audio sensitivity, and transcription robustness vary independently, so voice AI should be evaluated as a multidimensional...

  7. Can Large Audio Language Models Ignore Multilingual Distractors? An Evaluation of Their Selective Auditory Attention Capabilities

    eess.AS 2026-05 unverdicted novelty 6.0 of 10

    Introduces the MUSA benchmark and evaluates LALMs showing that strong single-speaker performance fails to ensure robust selective attention under multilingual interference, with errors from source confusion and unreso...

  8. AudioMosaic: Contrastive Masked Audio Representation Learning

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    AudioMosaic learns general-purpose audio representations through contrastive pre-training with structured spectrogram masking, reaching state-of-the-art results on standard benchmarks and improving audio-language tasks.

  9. VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech

    eess.AS 2026-04 unverdicted novelty 6.0 of 10

    VIBE evaluates generative biases in large audio-language models with real-world speech and open-ended tasks, showing that gender cues produce larger distributional shifts than accent cues across 11 tested models.

  10. Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics

    cs.CV 2026-01 conditional novelty 6.0 of 10

    Prototypicality bias: common text-to-image metrics systematically prefer plausible-but-wrong images over correct non-prototypical ones; PROTOSCORE mitigates but does not eliminate the failure.

  11. AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs

    cs.SD 2025-09 unverdicted novelty 6.0 of 10

    AU-Harness introduces an efficient unified evaluation framework for audio LLMs featuring batch optimizations, multi-turn dialogue support, and standardized protocols for fair comparisons.

Reference graph

Works this paper leans on

75 extracted references · 57 canonical work pages · cited by 10 Pith papers

  1. [38]

    V oice jailbreak attacks against GPT-4o, 2024

    Xinyue Shen, Yixin Wu, Michael Backes, and Yang Zhang. V oice jailbreak attacks against GPT-4o, 2024

    work page 2024

  2. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [2]

    The Claude 3 model family: Opus, Sonnet, Haiku, 2024

    Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku, 2024

    work page 2024

  4. [3]

    Common voice: A massively-multilingual speech corpus

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019

    arXiv 1912

  5. [4]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  6. [5]

    Towards multimodal sarcasm detection (an _obviously_ perfect paper)

    Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an _obviously_ perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), Florence, Italy, 7 2019. Association for Computational Linguistics

    work page 2019

  7. [6]

    Qwen2-audio technical report

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuan- jun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024

    arXiv 2024

  8. [7]

    V oxceleb2: Deep speaker recognition

    Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. V oxceleb2: Deep speaker recognition. arXiv preprint arXiv:1806.05622, 2018

    arXiv 2018

Show all 75 references
  1. [8]

    Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit

    Jacob Cohen. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213, 1968

    1968

  2. [9]

    FLEURS: Few-shot learning evaluation of universal representations of speech

    Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. FLEURS: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805. IEEE, 2023

    2022

  3. [10]

    MuTox: Uni- versal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detector

    Marta R Costa-jussà, Mariano Coria Meglioli, Pierre Andrews, David Dale, Prangthip Hansanti, Elahe Kalbassi, Alex Mourachko, Christophe Ropers, and Carleigh Wood. MuTox: Uni- versal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detector. arXiv preprint arXiv:2401.05060, 2024

    2024 arXiv

  4. [11]

    Speech-transformer: a no-recurrence sequence-to- sequence model for speech recognition

    Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformer: a no-recurrence sequence-to- sequence model for speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 5884–5888. IEEE, 2018

    2018

  5. [12]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  6. [13]

    Alpacafarm: A simulation framework for methods that learn from human feedback

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems , 36...

    2023

  7. [14]

    Clap learning audio concepts from natural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5. IEEE, 2023

    2023

  8. [15]

    Examining gender and racial bias in large vision- language models using a novel dataset of parallel images

    Kathleen C Fraser and Svetlana Kiritchenko. Examining gender and racial bias in large vision- language models using a novel dataset of parallel images. arXiv preprint arXiv:2402.05779, 2024

    2024 arXiv

  9. [16]

    CSR-I (WSJ0) Complete

    John S Garofolo, David Graff, Doug Paul, and David Pallett. CSR-I (WSJ0) Complete. 2007

    2007

  10. [17]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  11. [18]

    Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities

    Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. In Proceedings of the 2024 Conf...

    2024

  12. [19]

    V ocalsound: A dataset for improving human vocal sounds recognition

    Yuan Gong, Jin Yu, and James Glass. V ocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 151–155. IEEE, 2022

    2022

  13. [20]

    Sequence transduction with recurrent neural networks

    Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.3711, 2012

    2012 arXiv

  14. [21]

    Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks

    Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pages 369–376, 2006

    2006

  15. [22]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  16. [23]

    Design of a linguistic statistical decoder for the recognition of continuous speech

    Frederick Jelinek, Lalit Bahl, and Robert Mercer. Design of a linguistic statistical decoder for the recognition of continuous speech. IEEE Transactions on Information Theory , 21(3):250–256, 1975

    1975

  17. [24]

    Gemini 2.5: Our most intelligent AI model

    Koray Kavukcuoglu. Gemini 2.5: Our most intelligent AI model. 03 2025

    2025

  18. [25]

    AudioCaps: Generat- ing captions for audios in the wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generat- ing captions for audios in the wild. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 1...

    2019

  19. [26]

    Prometheus-vision: Vision-language model as a judge for fine-grained evaluation

    Seongyun Lee, Seungone Kim, Sue Park, Geewook Kim, and Minjoon Seo. Prometheus-vision: Vision-language model as a judge for fine-grained evaluation. In Findings of the Association for Computational Linguistics ACL 2024, pages 11286–11315, 2024

    2024

  20. [27]

    Vhelm: A holistic evaluation of vision language models

    Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, et al. Vhelm: A holistic evaluation of vision language models. Advances in Neural Information Processing Systems , 37:140632– 140666, 2024

    2024

  21. [28]

    Holistic evaluation of text-to-image models

    Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models. In Advances in Neural Information Processing Systems , volume 36, 2024

    2024

  22. [29]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Aco...

    2023

  23. [30]

    The next chapter of the Gemini era for developers

    Shrestha Basu Mallic and Kathy Korevec. The next chapter of the Gemini era for developers. 12 2024

    2024

  24. [31]

    Hello GPT-4o, 2024

    OpenAI. Hello GPT-4o, 2024

    2024

  25. [32]

    Introducing our next-generation audio models, Mar 2025

    OpenAI. Introducing our next-generation audio models, Mar 2025. Blog post

    2025

  26. [33]

    LibriSpeech: an ASR corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. LibriSpeech: an ASR corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 5206–5210. IEEE, 2015

    2015

  27. [34]

    MELD: A multimodal multi-party dataset for emotion recognition in conversa- tions

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal multi-party dataset for emotion recognition in conversa- tions. In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meet...

    2019

  28. [35]

    Mls: A large-scale multilingual dataset for speech research

    Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411, 2020. 12

    2012 arXiv

  29. [36]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  30. [37]

    Speech robust bench: a robustness benchmark for speech recognition

    Muhammad A Shah, David Solans Noguero, Mikko A Heikkila, Bhiksha Raj, and Nicolas Kourtellis. Speech robust bench: a robustness benchmark for speech recognition. arXiv preprint arXiv:2403.07937, 2024

    2024 arXiv

  31. [39]

    Salmonn: Towards generic hearing abilities for large language models

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, MA Zejun, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations , 2024

    2024

  32. [40]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  33. [41]

    CoV oST 2 and massively multilingual speech-to-text translation

    Changhan Wang, Anne Wu, and Juan Pino. CoV oST 2 and massively multilingual speech-to-text translation. arXiv preprint arXiv:2007.10310, 2020

    2007 arXiv

  34. [42]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215, 2025

    2025 arXiv

  35. [43]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  36. [44]

    AIR-Bench: Benchmarking large audio-language models via generative comprehension

    Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. AIR-Bench: Benchmarking large audio-language models via generative comprehension. arXiv preprint arXiv:2402.07729, 2024

    2024 arXiv

  37. [45]

    Speechlm: Enhanced speech pre-training with unpaired textual data

    Ziqiang Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, et al. Speechlm: Enhanced speech pre-training with unpaired textual data. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2024

    2024

  38. [46]

    6-12" #Children

    Shiyu Zhou, Linhao Dong, Shuang Xu, and Bo Xu. Syllable-based sequence-to-sequence speech recognition with the transformer in mandarin chinese. arXiv preprint arXiv:1804.10752, 2018. 13 A Aspect coverage Table A1: Models and aspects evaluated prior to AHELM, compiled to the be...

    2018 arXiv

  39. [47]

    ‘Answer’ agrees with ‘Groundtruth’

  40. [48]

    ‘Answer’ is a logical inference from ‘Question’

  41. [49]

    humorous and imaginative

    There is no ambiguity when answering ‘Question’ with ‘Answer’. Output only ‘yes’ or ‘no’. Do not explain. Context: {question} Answer: {validator_answer} Groundtruth: {groundtruth} Figure A9: Prompt used in the matching of the answer between the validator and the ground-truth (...

  42. [50]

    We obtain a list of contrasting roles

  43. [51]

    We generate transcripts of utterances that could be spoken by by both roles using an LM

  44. [52]

    E.1.1 Obtaining a list of contrasting roles We use the list of roles from PAIRS (replicated in Table A4) to seed the generation of speech content

    We generate audio speech using text-to-speech engines. E.1.1 Obtaining a list of contrasting roles We use the list of roles from PAIRS (replicated in Table A4) to seed the generation of speech content. In the current iteration of PARADE, we do not explore new roles or categori...

  45. [53]

    Write a one-sentence feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general

  46. [54]

    You should refer to the score rubric

    After writing the one-sentence feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric

  47. [55]

    Please do not generate any other opening, closing, and explanations. ###The ground truth answer: {{ground_truth}} ###Model Response to evaluate: {{orig_response}} ###Score Rubrics: [Does the predicted response align with the ground truth in terms of accuracy?] Score 1: The res...

    2024

  48. [56]

    haha”) or throat clearing (e.g., “ahem

    The ‘transcribe + LM’ paradigm falls short in more ‘natural’ tasks. Comparing the dedicated ASR models, we observe that GPT-4o Transcribe and GPT-4o Mini Transcribe fail to transcribe properly when fed speech in more ‘natural’ settings. For example, in MUStARD, where the audio...

  49. [57]

    Gemini and baselines perform well on multilinguality but performances are skewed towards internet data distribution. The baseline systems and the Gemini models dominate the top half of the multilinguality leaderboard, with GPT-4o Transcribe + GPT-4o (2024-11-20) performing the...

    2024

  50. [58]

    From Table A9, we see that Qwen2-Audio Instruct takes the lead in audio knowledge, followed by Gemini 2.5 Pro (05-06 Preview) and then Gemini 2.0 Flash

    Open-weight models can compete head-to-head with the best closed-API models on audio knowledge. From Table A9, we see that Qwen2-Audio Instruct takes the lead in audio knowledge, followed by Gemini 2.5 Pro (05-06 Preview) and then Gemini 2.0 Flash. The baseline systems score w...

  51. [59]

    When looking at the safety aspect, we see that OpenAI models are robust to the voice jailbreak attack

    OpenAI’s models are better at defending against jailbreak attacks. When looking at the safety aspect, we see that OpenAI models are robust to the voice jailbreak attack. It may be possible that this vulnerability has specifically been patched by OpenAI since the original paper...

  52. [60]

    We explain our benchmark in Section 3 and describe the experiments in Section 4 and report results in Section 5

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We present our thesis in Section 1. We explain our benchmark in Section 3 and describe the experiments in Section 4 an...

  53. [61]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss limitations in Section 6.1 Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has...

  54. [62]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] 46 Justification: See Section 6.2. Guidelines: • The answer NA means that the paper does not include theore...

  55. [63]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  56. [64]

    com/stanford-crfm/helm and the new datasets at https://huggingface.co/ datasets/UCSC-VLAA/PARADE_audio and https://huggingface.co/datasets/ stanford-crfm/CoReBench_v1

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 47 Answer: [Yes] Justification: We provide the exact cod...

  57. [65]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: See Section 4. Guidelines: • ...

  58. [66]

    But we do not compute error bars for other scenarios

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We report the p-value for the fairness aspect on L...

  59. [67]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We state the number of ...

  60. [68]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We followed the NeurIPS Code of Ethics and discussed our limitations in Section...

  61. [69]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: See Section 6.1. Guidelines: • The answer NA means that there is no societal impact of the work performed...

  62. [70]

    Before transforming tran- scripts to audio, we performed human scrutiny of the audio transcripts to make sure that there is no improper or toxic content in the metadata

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: In this paper,...

  63. [71]

    We cite all the datasets and models used in our work

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All the models u...

  64. [72]

    PARADE is available at https://huggingface.co/datasets/UCSC-VLAA/PARADE_ audio

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We provide executable codes in our codebase: https://github.com/ stanford-crfm/helm. PARADE is available at https://...

  65. [73]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  66. [74]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  67. [75]

    Answer: [Yes] Justification: As detailed in the Appendix B, we leverage OpenAI’s GPT-4o to create audio transcripts for the curation of PARADE benchmark

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.