Pith. sign in

REVIEW 5 major objections 6 minor 37 references

ORCA predicts both the expected correctness and the disagreement of human judgments for open-ended audio question answering, using a Beta distribution over ratings, and a small text-only model matches or beats large LLM judges while providi

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 19:33 UTC pith:3WAUBHXK

load-bearing objection The dataset and Beta-judge idea are real, but the abstract overclaims and the headline numbers rest on human-corrected rationales; the paper needs revision before its practical claims hold. the 5 major comments →

arxiv 2512.09066 v2 pith:3WAUBHXK submitted 2025-11-28 cs.SD cs.AIcs.CL

ORCA: Open-ended Response Correctness Assessment for Audio Question Answering

\v{S}imon Sedl\'a\v{c}ek , Sara Barahona , Bolaji Yusuf , Laura Herrera-Alarc\'on , Santosh Kesiraju , Cecilia Bola\~nos , Alicia Lozano-Diez , Sathvik Udupa
show 4 more authors
Fernando L\'opez Allison Ferner Ramani Duraiswami Jan \v{C}ernock\'y
This is my paper
classification cs.SD cs.AIcs.CL
keywords audio question answeringanswer correctness assessmentLLM-as-a-judgeBeta distributionuncertainty quantificationannotation disagreementLALM evaluationopen-ended response evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ORCA is a lightweight, text-only evaluator for open-ended audio question answering that predicts the full distribution of human correctness ratings, not just the average. It models ratings with a Beta distribution, outputting both the expected score and the variance, so it can tell consensus from genuine disagreement. Trained on human annotations collected through a three-stage pipeline with structured feedback and expert correction, ORCA reaches a Spearman correlation of about 0.91 with average human scores on unseen questions, matching or beating much larger LLM judges while requiring a single forward pass. The paper argues that this makes audio QA evaluation cheaper, more reproducible, and able to flag ambiguous or flawed benchmark items instead of hiding them in a mean score.

Core claim

On the paper's own terms, ORCA establishes that answer correctness for open-ended audio QA can be assessed reliably by a small text-only model—fed the question, reference answer, a machine-generated rationale, and a speech transcript—when the model is trained to predict a Beta distribution over human ratings. The model outputs log α and log β, whose mean and variance match human average scores and human disagreement. In experiments on two audio QA benchmarks with ratings from 15 different audio language models, the best ORCA configuration reaches a Spearman correlation of 0.91 with mean human judgments on unseen-question splits and also achieves low mean-absolute error on predicted variance,

What carries the argument

The central object is the Beta distribution over normalized correctness ratings. A small pre-trained transformer encodes the concatenated text inputs, and an MLP head predicts log α and log β. The mean α/(α+β) serves as the point estimate of correctness, while the variance αβ/((α+β)^2(α+β+1)) serves as the uncertainty estimate. Training maximizes the log-likelihood of each individual human rating, treating every rating as a sample from the predicted distribution. A clamping post-processing step sharpens predictions that are near zero or one and have low predicted variance.

Load-bearing premise

The whole evaluation—for annotators, ORCA, and LLM judges—rests on the assumption that the text-only context (question, reference answer, machine-generated rationale, and transcript) carries enough of the audio's content for correctness to be judged; the paper itself acknowledges in Stage 1a that automatically generated rationales can be non-informative or based on textual cues rather than on the audio.

What would settle it

Construct a set of audio QA items where the audio contains decisive information that the transcript and rationale systematically omit or misrepresent—for example, a sound event mislabeled in the rationale with no transcript available—and check whether ORCA's ratings and confidence track human ratings made with access to the audio. If ORCA's accuracy and calibration collapse on these items while human judges who can listen still agree, the text-only grounding premise fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Open-ended audio QA evaluation no longer requires an audio model as judge, avoiding the circularity of using audio-language models to rate other audio-language models.
  • Benchmark reports can include a per-item disagreement score, letting researchers identify ambiguous, poorly written, or under-specified questions rather than averaging over them.
  • A single forward pass through a small open-weight model replaces multi-step LLM judging, cutting computational cost and improving reproducibility.
  • The curated dataset of thousands of human ratings across many audio-language models can serve as a reusable evaluation resource for the community.
  • LLM-judge ratings, despite being noisier, can be used for pretraining, pointing toward reduced human annotation burden for future benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If ORCA's variance estimates are trustworthy, the same Beta-distribution head could be adapted to any evaluation or self-assessment task, letting models report calibrated confidence instead of just a score.
  • The three-stage human-AI refinement loop implicitly measures benchmark quality: the paper's correction statistics suggest existing audio QA benchmarks contain substantial shares of ambiguous or flawed questions, which distributional evaluation surfaces.
  • A natural extension the paper does not pursue is active learning: use ORCA's predicted variance to decide which items need additional human annotation, concentrating effort where disagreement is predicted to be high.
  • The text-only premise could be stress-tested on categories where audio carries information invisible to transcripts and rationales, such as emotional nuance or non-speech sound texture—the exact cases where human annotators most often disagree.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces ORCA, a text-only framework that models human correctness judgments for open-ended audio question answering as a Beta distribution, predicting both the mean and the variance (disagreement) of ratings. The authors collect human annotations through a three-stage pipeline: Gemini/Whisper-based context generation, human rating with structured feedback, and expert human-AI correction. They then train small open-weight language models (OLMo, Gemma, Llama) with an MLP head to predict Beta parameters from question, reference, rationale, transcript, and candidate answer text. Experiments on two audio QA benchmarks (MMAU test-mini and MMAR) report a best Spearman correlation of 0.9103 with mean human judgments for the clamped Gemma3-12B ORCA model, outperforming several LLM-judge baselines including Gemini-2.5-Flash, while also providing variance estimates that LLM judges do not. The paper claims that ORCA generalizes to unseen LALM response styles and identifies ambiguous benchmark items via predicted variance.

Significance. If substantiated, ORCA would be a practically valuable contribution: a lightweight, reproducible, open-weight evaluator for open-ended audio QA that goes beyond mean scores to capture annotator disagreement. The release of models, code, and curated annotations is a concrete benefit to the community, and the distributional modeling of human judgment variability is a timely direction. The method is conceptually simple and the training cost is low (≈15 minutes per run on a single GPU). The manuscript also includes useful input-ablation analysis and a study of training on LLM-judge versus human labels. However, the strength of these claims is currently undermined by several load-bearing gaps between the abstract, the reported evaluation protocol, and the data curation decisions, which must be addressed before the contribution can be accepted as stated.

major comments (5)
  1. [Abstract and §5.3–6.1] The abstract claims that ORCA models 'generalize to unseen benchmarks with a score of 0.85' and that the method employs 'curriculum learning'. Neither claim appears anywhere in the body. Section 5.3 defines only two evaluation scenarios—unseen questions and unseen LALMs—both within the same two corrected benchmarks (MMAU and MMAR). No third benchmark is used, and Table 2's best Spearman is 0.9103, not 0.85. The '0.85' number and curriculum learning are unsupported and must either be added with a concrete experimental definition or removed from the abstract.
  2. [§5.2 and Table 2] All headline numbers are computed on the 2,459 question-answer pairs that survived the Stage 3 filter, after post hoc removal of 1,121 pairs and 3,150 ratings (32% of the total). The justification that the removed ratings had lower agreement (α=0.59) is circular: the paper's own motivation is that genuinely ambiguous items produce low agreement. Removing them selectively discards the hard cases that ORCA is supposed to model. The paper should report results on the original 3,580 pairs, or otherwise treat the filtering as a modeling choice and ablate it, showing that ORCA still outperforms LLM judges without the filter.
  3. [§5.5 and Table 2] The clamping post-processing threshold is 'optimized on the development set to maximize ρ+τ−MAE_μ'. Since the same dev set is used for the reported results, the clamped ORCA numbers in Table 2 include a tuned free parameter. The paper should report unclamped results for all ORCA models, and clarify how the threshold would be set in a deployment scenario without access to human ratings. Without this, the comparison between clamped ORCA and the unclamped LLM judges is not parameter-free.
  4. [§3.1–§3.2 and §4] There is a circularity concern that is central to the claimed validity of the gold standard. Human annotators (and ORCA, and the LLM judges) all receive the same Gemini-generated rationale as part of the input. The paper itself notes that Gemini sometimes produces non-informative rationales and that 30.5% of questions were flagged as having insufficient rationale, leading to human corrections in Stage 3. However, the reported evaluation is run on the corrected rationales, and ORCA's input always includes that corrected rationale. Thus the high agreement between ORCA and human ratings may largely reflect shared reliance on the same textual rationale, rather than independent assessment of the audio. The paper should provide an ablation where ORCA is trained/evaluated without rationales (or with audio-only human ratings) to quantify how much of the reported correlation is attributable to th
  5. [§5.2, §5.6, and deployment claim] The paper motivates ORCA as a replacement for LLM judges in real-world audio QA evaluation, but the models are trained and tested on the post-correction dataset, in which rationales and reference answers were manually fixed by six domain experts. In deployment, ORCA would receive raw Gemini-generated rationales and uncorrected benchmark references. No experiment evaluates this setting. The authors should test ORCA on the pre-correction data (e.g., the original rationales before Stage 3) and report the performance drop, if any. Unless this is done, the claimed practical advantage over LLM judges is not established.
minor comments (6)
  1. [Abstract and Introduction] The abstract and Section 1 use different annotation counts: 11,721 vs. 11,721 in Section 5.2 is consistent, but the abstract says '9,663 annotations' in the reader's take? Actually the manuscript abstract in the provided text says 9,663; the body says 11,721. Check consistency: §5.2 reports 11,721 total annotations, while the abstract says 9,663. Please reconcile.
  2. [§5.2 and Table 1] Typo: 'Krippendorff' is misspelled as 'Krip-pendorff' in §2.2? Check reference. Also the citation format for Krippendorff (2019) is inconsistent in the text.
  3. [Figure 2] The figure label 'LLM-Judge Correctness score [1,5]' appears twice, once for humans and once for LLM judges. Please disambiguate the annotation source.
  4. [Appendix B] Typo in Figure 7 prompt: 'trasncript' should be 'transcript'. Also Appendix B.1 says 'Gemini-2.5-Flash' while Figure 6 says 'Gemini-2.0-Flash'—please align.
  5. [§6.2] The held-out LALM experiment uses only OLMo2-7B clamped, and Figure 3's caption says 'OLMo-7B' while the text says 'OLMo2-7B'. Please make model names consistent throughout.
  6. [§5.3] The two-level stratification description is clear, but it would help to report the number of unique questions in the test set of each split and the overlap of question categories across splits, to assess the difficulty of the unseen-question scenario.

Circularity Check

0 steps flagged

No circularity: ORCA is a supervised Beta-regression model, not a derivation from its inputs; the abstract's unsupported 0.85 claim is a reporting gap, not circularity.

full rationale

ORCA's derivation chain is a standard supervised-learning setup, not a circular one. The model's outputs (log alpha, log beta) are functions of the input text (q;r;a;t;c) via Eq. (1), and the training objective in Eq. (5) is maximum likelihood over human ratings y_{i,j} under the predicted Beta distribution. The target ratings are human judgments collected using the same textual context that ORCA later consumes; this shared context can create label-feature correlation, but it does not make a held-out prediction equivalent to its inputs by construction. The evaluation uses five stratified unseen-question splits and five held-out-LALM splits (Section 5.3), and the clamping threshold in Section 5.5 is optimized on the development set, not on the test labels. The only overlapping-author citation is MMAU-Pro (Kumar et al., 2025) in Related Work; it is not load-bearing, and no uniqueness theorem or prior-work ansatz is invoked to force the Beta choice. Separately, the abstract's 'generalize to unseen benchmarks with a score of 0.85' is not supported by any reported number in Sections 5-6 (the closest reported results are Table 2's 0.9103 on unseen-question splits); that is a reporting/validation gap, not a circularity. I therefore find no significant circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

No new physical entities are introduced. The load-bearing inputs are human ratings, Gemini/Whisper-generated text, and two post-hoc choices (data filtering and clamping threshold).

free parameters (2)
  • Post-hoc filtering threshold for unreliable annotations = 541 questions / 1,121 QA pairs / 3,150 ratings removed
    Ratings flagged via structured feedback were excluded after expert review; this improves alpha from 0.76 to 0.82 and defines the dataset on which all results are measured (Section 5.2).
  • Clamping variance threshold for ORCA post-processing = not reported (optimized on dev set)
    ORCA scores within 0.125 of 0/1 are hard-clamped when predicted variance is below a threshold tuned to maximize rho+tau-MAE on the development set (Section 5.5).
axioms (5)
  • domain assumption Human 1-5 correctness ratings are a valid ground truth for open-ended audio QA correctness
    Section 3.2 defines the rating scale and treats human judgments as gold standard.
  • domain assumption Gemini-generated rationales and Whisper transcripts preserve enough audio information for text-only correctness judgment
    Stage 1a constructs textual grounding; Section 2.1 claims this avoids circular dependency, but it assumes the rationale is accurate; paper admits Gemini sometimes generates non-informative rationales.
  • domain assumption Beta distribution can represent the distribution of human correctness ratings
    Section 4.2 motivates Beta for [0,1] ratings, but no goodness-of-fit check against empirical rating histograms is provided.
  • ad hoc to paper Post-hoc exclusion of 32% of ratings does not bias the corrected dataset
    Section 5.2 filters 541 questions; the paper argues low agreement (0.59) confirms unreliability, but this is a post hoc judgment that shapes every reported metric.
  • domain assumption A single MLP on the final hidden state of a pre-trained LLM is sufficient to map textual grounding to Beta parameters
    Section 4.1 architecture; no analysis of failure modes for long or out-of-distribution responses.

pith-pipeline@v1.3.0-alltime-deepseek · 15005 in / 16135 out tokens · 153908 ms · 2026-08-03T19:33:31.266460+00:00 · methodology

0 comments
read the original abstract

Reliable assessment of the abilities of large audio language models (LALMs) is essential to advancing the state of the art. As benchmarks rapidly evolve to incorporate complex reasoning and subjective tasks, they increasingly necessitate open-ended responses from LALMs. We present Open-ended Response Correctness Assessment (ORCA) -- a reliable and lightweight model-based approach for answer correctness and disagreement modeling. We employ a three-stage annotation pipeline combining human judgment, structured feedback, and human-AI correction, yielding 9,663 annotations across 3,699 question-answer pairs from 15 LALMs on three audio understanding and reasoning benchmarks (achieving a Krippendorff's alpha of 0.82). Our experiments employing curriculum learning show that ORCA models achieve a Spearman correlation of 0.91 with average human correctness ratings on seen benchmarks and generalize to unseen benchmarks with a score of 0.85, outperforming several LLM judge baselines including Gemini 2.5 Flash. Furthermore, we demonstrate that ORCA's predicted variance correlates strongly with human disagreement, allowing it to effectively identify problematic benchmark items.

Figures

Figures reproduced from arXiv: 2512.09066 by Alicia Lozano-Diez, Allison Ferner, Bolaji Yusuf, Cecilia Bola\~nos, Fernando L\'opez, Jan \v{C}ernock\'y, Laura Herrera-Alarc\'on, Ramani Duraiswami, Santosh Kesiraju, Sara Barahona, Sathvik Udupa, \v{S}imon Sedl\'a\v{c}ek.

Figure 1
Figure 1. Figure 1: Two examples with similar mean ratings (µ ≈ 3) but different variances: low variance (top) in￾dicates consensus; high variance (bottom) reveals dis￾agreement. in which metrics assess the correctness of the an￾swer augmented by textual grounding (rationales and transcriptions) (Yang et al., 2024), avoiding the circular dependency of using audio models to judge audio models. This framework applies uni￾formly… view at source ↗
Figure 2
Figure 2. Figure 2: Annotation framework pipeline. Data preparation (Stage 1) generates rationales via Gemini, transcripts [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of LLM-judge (Gemini, offline LLM-judge fusion) to clamped ORCA OLMo-7B when [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study on ORCA and avg. LLM￾judge inputs. We report MAE of the predicted scores with respect to the average human rating. Q, R, T de￾note the original question, the rationale, and the tran￾script, respectively. Training data Spearman ρ MAEµ (1) Human data 0.899 ± 0.010 0.083 ± 0.004 (2) LLM-judge 0.886 ± 0.003 0.108 ± 0.006 (3) LLM-judge (all data) 0.889 ± 0.010 0.103 ± 0.014 (4) Human + LLM-judge … view at source ↗
Figure 5
Figure 5. Figure 5: Histogram of human ratings for answer correctness before and after correction (Stage 3). [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prompt used for generating rationales with Gemini-2.0-Flash. The model receives the audio file along [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Full prompt formulation for the LLM-judges evaluation. Other variants were generated by removing [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Modified full LLM-judge prompt example. Base variants omit the transcript, rationale, and the question. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 2 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. 2025. https://doi.org/10.18653/v1/2025.acl-long.169 Which of these best describes multiple choice evaluation with LLM s? a) forced B ) flawed C ) fixable D ) all of the above . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    doi:10.18653/v1/2025.acl-long.169 2025

  4. [4]

    Debarpan Bhattacharya, Apoorva Kulkarni, and Sriram Ganapathy. 2025. https://doi.org/10.21437/Interspeech.2025-1228 Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning . In Interspeech 2025 , pages 2068--2072

    work page doi:10.21437/interspeech.2025-1228 2025

  5. [5]

    Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin B \"o rschinger, and Tal Schuster. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.20 Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 291--305, Abu Dhabi, ...

    doi:10.18653/v1/2022.emnlp-main.20 2022

  6. [6]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. http://arxiv.org/abs/2407.10759 Qwen2-audio technical report

    Pith/arXiv arXiv 2024

  7. [7]

    Ding Ding, Zeqian Ju, and Yichong Leng. 2025. https://arxiv.org/abs/2504.18425 Kimi- Audio Technical Report . Technical report, MoonshotAI

    Pith/arXiv arXiv 2025

  8. [8]

    Gemini Team . 2025. http://arxiv.org/abs/2507.06261 Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Pith/arXiv arXiv 2025

  9. [9]

    Gemma Team . 2025 a . http://arxiv.org/abs/2503.19786 Gemma 3 Technical Report

    Pith/arXiv arXiv 2025

  10. [10]

    Gemma Team . 2025 b . https://ai.google.dev/gemma/docs/gemma-3n Gemma 3n

    2025

  11. [11]

    Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro

    Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S. Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro. 2025. https://openreview.net/pdf?id=xWu5qpDK6U Audio Flamingo 2: An Audio - Language Model with Long - Audio Understanding and Expert Reasoning Abilities . In Proceedings of the 42 nd International Conference on Machine Learning. PMLR

    2025

  12. [12]

    Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.361 GAMA : A large audio-language model with advanced audio understanding and complex reasoning abilities . In Proceedings of the 2024 Conference on Empirical Meth...

    doi:10.18653/v1/2024.emnlp-main.361 2024

  13. [13]

    Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. 2025. http://arxiv.org/abs/2507.08128 Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models . In NeurIPS

    Pith/arXiv arXiv 2025

  14. [14]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.248 Prometheus 2: An open source language model specialized in evaluating other language models . In Proceedings of the 2024 Conference on Empirical Methods in Na...

    doi:10.18653/v1/2024.emnlp-main.248 2024

  15. [15]

    Klaus Krippendorff. 2019. https://doi.org/10.4135/9781071878781 Content Analysis : An Introduction to Its Methodology . SAGE Publications, Inc

    doi:10.4135/9781071878781 2019

  16. [16]

    Sonal Kumar, Šimon Sedláček, Vaibhavi Lokegaonkar, Fernando López, Wenyi Yu, Nishit Anand, Hyeonggon Ryu, Lichang Chen, Maxim Plička, Miroslav Hlaváček, William Fineas Ellingwood, Sathvik Udupa, Siyuan Hou, Allison Ferner, Sara Barahona, Cecilia Bolaños, Satish Rahi, Laura Herrera-Alarcón, Satvik Dixit, Siddhi Patil, Soham Deshmukh, Lasha Koroshinadze, Ya...

    Pith/arXiv arXiv 2025

  17. [17]

    Elisa Leonardelli, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, Alexandra Uma, and Massimo Poesio. 2023. https://doi.org/10.18653/v1/2023.semeval-1.314 S em E val-2023 task 11: Learning with disagreements ( L e W i D i) . In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval...

    doi:10.18653/v1/2023.semeval-1.314 2023

  18. [18]

    Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, and Yejin Choi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.51 We ' re afraid language models aren ' t modeling ambiguity . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 790--807, Singap...

    doi:10.18653/v1/2023.emnlp-main.51 2023

  19. [19]

    Llama Team . 2024. http://arxiv.org/abs/2407.21783 The Llama 3 Herd of Models

    Pith/arXiv arXiv 2024

  20. [20]

    Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, and Hung-yi Lee. 2025 a . https://doi.org/10.48550/arXiv.2409.20007 DeSTA2 : Developing Instruction - Following Speech Language Model Without Speech Instruction - Tuning Data . ArXiv:2409.20007 [eess]

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.20007 2025

  21. [21]

    Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Chih-Kai Yang, Chee-En Yu, Chun-Wei Chen, Wei-Chih Chen, Chien-yu Huang, Yi-Cheng Lin, Yu-Xiang Lin, Chi-An Fu, Chun-Yi Kuan, Wenze Ren, Xuanjun Chen, Wei-Ping Huang, En-Pei Hu, Tzu-Quan Lin, Yuan-Kuei Wu, Kuan-Po Huang, Hsiao-Ying Huang, Huang-Cheng Chou, Kai-Wei Chang, Cheng-Han C...

    doi:10.48550/arxiv.2507.02768 2025

  22. [22]

    Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, et al. 2025. Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix. arXiv preprint arXiv:2505.13032

    Pith/arXiv arXiv 2025

  23. [23]

    Grigor Nalbandyan, Rima Shahbazyan, and Evelina Bakhturina. 2025. https://doi.org/10.18653/v1/2025.naacl-industry.39 SCORE : Systematic CO nsistency and robustness evaluation for large language models . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies...

    doi:10.18653/v1/2025.naacl-industry.39 2025

  24. [24]

    Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. 2022. https://doi.org/10.18653/v1/2022.emnlp-demos.33 POTATO : The portable text annotation tool . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 327--337, Abu...

    doi:10.18653/v1/2022.emnlp-demos.33 2022

  25. [25]

    Barbara Plank. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.731 The ``problem'' of human label variation: On ground truth in data, modeling and evaluation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671--10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics

    doi:10.18653/v1/2022.emnlp-main.731 2022

  26. [26]

    Qwen Team . 2025. http://arxiv.org/abs/2412.15115 Qwen2.5 Technical Report

    Pith/arXiv arXiv 2025

  27. [27]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. https://proceedings.mlr.press/v202/radford23a.html Robust speech recognition via large-scale weak supervision . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28492--28518. PMLR

    2023

  28. [28]

    Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha

    S. Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2024. https://openreview.net/forum?id=TeVAZXr3yv MMAU : A Massive Multi - Task Audio Understanding and Reasoning Benchmark . In The Twelfth International Conference on Learning Representations

    2024

  29. [29]

    Marta Sandri, Elisa Leonardelli, Sara Tonelli, and Elisabetta Jezek. 2023. https://doi.org/10.18653/v1/2023.eacl-main.178 Why don ' t you do it right? analysing annotators' disagreement in subjective tasks . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2428--2441, Dubrovnik, Croatia....

    doi:10.18653/v1/2023.eacl-main.178 2023

  30. [30]

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. https://openreview.net/forum?id=14rn7HpKVk SALMONN : Towards Generic Hearing Abilities for Large Language Models . In The Twelfth International Conference on Learning Representations

    2024

  31. [31]

    Smith, and Hannaneh Hajishirzi

    Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal Guerquin, David Heineman, Hamish Ivison, Pang Wei Koh, Jiacheng...

    2025

  32. [32]

    Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F. Chen. 2025. https://doi.org/10.18653/v1/2025.naacl-long.218 A udio B ench: A universal benchmark for audio large language models . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lingui...

    doi:10.18653/v1/2025.naacl-long.218 2025

  33. [33]

    Wen Wu, Wenlin Chen, Chao Zhang, and Phil Woodland. 2024. https://doi.org/10.18653/v1/2024.findings-acl.67 Modelling variability in human annotator simulation . In Findings of the Association for Computational Linguistics: ACL 2024, pages 1139--1157, Bangkok, Thailand. Association for Computational Linguistics

    work page doi:10.18653/v1/2024.findings-acl.67 2024

  34. [34]

    Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. 2025. http://arxiv.org/abs/2503.02318 Audio-reasoner: Improving reasoning capability in large audio language models

    arXiv 2025

  35. [35]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. http://arxiv.org/abs/2503.20215 Qwen2.5-omni technical report

    Pith/arXiv arXiv 2025

  36. [36]

    Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. 2024. https://doi.org/10.18653/v1/2024.acl-long.109 AIR -bench: Benchmarking large audio-language models via generative comprehension . In Proceedings of the 62nd Annual Meeting of the Association for Computational Ling...

    doi:10.18653/v1/2024.acl-long.109 2024

  37. [37]

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 2024. http://arxiv.org/abs/2412.02612 Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot

    Pith/arXiv arXiv 2024