Pith. sign in

REVIEW 4 major objections 6 minor 53 references

Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that audio-language-model judges can pass contrastive matching yet fail native one-context judgments, so accuracy alone mis-certifies them.

desk verdict A careful audit with a real central finding: audio-judge models can solve pairwise contrasts but collapse in native one-context judgment; the main caveat is thin human validation for the single-turn task. read the letter →

arxiv 2608.06718 v2 pith:LJTPDE2X submitted 2026-08-07 cs.CL

Kevin Miller , Arjun Chandra , Venkatesh Saligrama This is my paper · ORCID
classification cs.CL
keywords counterfactualauditaudio-languagemodeljudgesparalinguisticreasoningresponseevaluationpointwisevspairwisejudgmentdiagnosticstatedecompositionspeech-to-speechPotemkinfailure
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that current audio-language model judges, when asked to pick the response appropriate to how a user sounded, frequently fail despite appearing capable. It introduces counterfactual audits: each item holds the transcript fixed and varies only affect, prosody, or the timing of an affective shift, so any valid judge must use the audio cue rather than lexical content. The headline result is that contrastive success overstates native judge reliability—Gemini models often match the two counterfactual worlds when both are shown, but collapse toward chance when only one audio context is given. The paper also shows that two judges with similar aggregate accuracy can fail in different places, which becomes visible through per-item diagnostic states. The upshot defended is that ALM judges should not be certified by end-to-end accuracy alone; deployment requires behavioral audits.

What carries the argument

The central object is the counterfactual audit item, a tuple holding a fixed transcript or conversation history, two audio renderings of the same words, and two responses appropriate to each rendering. The load-bearing comparison is between pointwise native judgment, where the judge sees one audio context and two responses, and pairwise contrastive recoverability, where both contexts and both responses are visible and must be matched. Each item is then decomposed into probes for perception, oracle response-mapping, and native judgment, giving an eight-state diagnostic whose mass distribution separates perception bottlenecks, response-mapping bottlenecks, Potemkin orchestration failures, shortcut successes, and reliable integrated judgment.

What would settle it

Run a larger human validation on all single-turn pointwise items under the hard cue: if a substantial fraction of the items that Gemini fails are also answered at chance by a panel of native English listeners, the pointwise collapse could reflect item ambiguity rather than model failure, whereas reliable human accuracy on the same items would confirm the collapse as a genuine judge limitation.

Watch

Extended reading notes

Core claim

The central discovery is a protocol gap: a judge can recover a counterfactual contrast in a pairwise setting and still be unreliable in the native pointwise setting. Concretely, Gemini-3-Pro reaches 91.0% pairwise accuracy on single-turn emotion items with a hard cue but only 65.3% pointwise; Gemini-2.5-Pro drops from 86.0% to 60.1%. In the positional multi-turn task the same models hover near chance pointwise even when they solve the pairwise matching with a transition cue (Gemini-3-Flash: 79.4% pairwise vs 53.0% pointwise). The diagnostic decomposition into perception, oracle response-mapping, and native judgment further shows that similar aggregate accuracies hide different failure modes: Gemini models often pass perception and oracle response-mapping yet fail native judgment, which the paper calls a Potemkin failure, while GPT models show earlier perception bottlenecks. The authors state the conclusion directly: a model may distinguish the two counterfactuals when both are shown, but fail when the same audio cue must control a single-context decision.

Load-bearing premise

The audit labels are assumed to be solvable by careful listeners, so that near-chance model accuracy indicates model failure rather than item ambiguity; this rests on only five annotators per task, with single-turn accuracy ranging from 62% to 100%.

Editorial extensions

If this is right

  • An ALM judge that scores well on pairwise or contrastive evaluations should not be assumed reliable for single-context deployment; the pairwise–pointwise gap should be reported as a separate metric.
  • Aggregate accuracy is insufficient as a certification metric; judges should be audited at the component level to identify whether failures are perceptual, mapping, or orchestration.
  • For temporal-causal paralinguistic judgment, current ALM judges are especially brittle: near-chance pointwise performance on positional-emotion shows they cannot reliably localize and use an affective shift.
  • Even models with strong component skills can be unreliable end-to-end, since Gemini models often pass perception and oracle response-mapping but fail native audio judgment, so scaffolding such as explicit cues may be needed.
  • The findings extend to real speech: the synthetic-to-real comparison yields highly correlated diagnostic profiles, suggesting the failure modes are not simply artifacts of synthesized audio.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pointwise collapse reflects a genuine competence–deployment gap, then a natural testable extension is to fine-tune ALMs on pointwise paralinguistic judgment using pairwise or oracle supervision; the diagnostic state distribution predicts which intervention—perception training, response-mapping training, or orchestration training—should help.
  • A consequence the paper does not draw is that the same audit could be applied to human listeners, since the five-annotator validation showed wide single-turn variation; reporting diagnostic states for humans would clarify whether annotator disagreement is perception-level or response-mapping-level.
  • Another implicit extension is to use the state masses as a regression-testing signal during deployment: monitoring the Potemkin failure state across model versions could detect when an audio upgrade fails to integrate into actual decisions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces counterfactual audits for evaluating audio-language models (ALMs) used as judges of spoken interactions. Each audit item fixes the transcript while varying the paralinguistic realization (affect, prosody, or the timing of an affective shift), forcing a valid judge to use the audio cue. Judges are evaluated in a native pointwise one-context protocol and a contrastive pairwise recoverability control, and each item is decomposed into three probes: perception (P), oracle response-mapping (O), and native judgment (J). Across Gemini, GPT, and open audio models, the authors report that Gemini models often achieve high pairwise accuracy (up to 91.0%) but near-chance pointwise accuracy (as low as 52.1%), while most other models remain near chance in both protocols. The diagnostic state analysis shows that similar aggregate accuracies can hide different failure modes, including 'Potemkin' failures where perception and response-mapping succeed but the native judgment fails. Robustness checks include generalization to real speech, generator-substitution, and response-quality balancing. The paper concludes that ALM judges should not be certified by accuracy alone.

Significance. If the results hold, this is a valuable methodological contribution to the evaluation of audio-language models: it provides a reusable instrument-auditing framework with a diagnostic state space that separates perception, response-mapping, and orchestration failures. The empirical finding that contrastive success can overstate native reliability is important for the growing use of ALMs as judges and reward models. The paper includes careful uncertainty reporting (Wilson CIs, paired bootstrap), multiple robustness checks, and a transparent limitations section. The promise of code release will aid reproducibility. The main risk is the load-bearing assumption that single-turn pointwise items are solvable by careful listeners, which is supported only by thin human validation.

major comments (4)
  1. [Section 4.3 / Appendix E] The single-turn pointwise solvability of the audit items is not adequately established. Only five annotators each judged a random subset of 50 items, with accuracy ranging from 62% to 100%, and no per-item agreement is reported. Since the headline single-turn pointwise collapse in Table 3 interprets near-chance model accuracy as a failure to use paralinguistic evidence, item ambiguity would directly inflate the measured protocol gap. Please provide per-item human labels or majority-vote reliability (e.g., per-item agreement or kappa) on a larger sample, or re-analyze the headline results on a subset of items with high human agreement. This is load-bearing for the Section 5.1 claim that the relevant contrast is recoverable but not deployed in the native setting.
  2. [Section 5.2 / Figure 3] The diagnostic state masses π_poj are reported without uncertainty, and the marginal P and O probes are near chance for many models (e.g., GPT-4o-mini P=50.5% in Table 14). The joint state distribution is therefore potentially dominated by probe noise, so the 'Potemkin' mass and the claimed differences in failure modes between judges may not be statistically meaningful. Please report bootstrap confidence intervals for the state masses, or a sensitivity analysis under chance-corrected scoring of the P and O probes.
  3. [Table 7 / Section 4.1] The sample sizes for single-turn pairwise evaluations are inconsistent with the stated 189 pairwise items (e.g., Gemini-2.5-Pro Pairwise Hard n=93; Gemini-3-Pro Pairwise No n=185). The paper should explain the exclusions (e.g., response-parsing failures, API errors) and confirm that the same item subset is used for all judges; otherwise the paired bootstrap protocol-gap comparisons in Table 6 may be computed over different item subsets for different judges.
  4. [Section 5.3 / Table 4] Using claude-haiku-4-5 as the LLM judge to rule out lexical-quality confounds introduces a circularity risk: the audit is designed to scrutinize LLM judges, yet another LLM judge is used to validate the absence of confounds. While this is a secondary robustness check, the paper should either use human ratings for the balance check or provide evidence that claude-haiku-4-5's quality judgments are themselves validated against human judgments.
minor comments (6)
  1. [Section 2] The term 'Potemkin' is used throughout; please define it explicitly at first use (the reference to Mancoridis et al. 2025 is helpful).
  2. [Appendix E, Table 21] For the single-turn task, the Accuracy (%) column should clarify that the 62-100% range is per-annotator accuracy on a random 50-item subset, not per-item agreement; consider reporting the distribution of item-level agreement as well.
  3. [Appendix B, Listings 1 and 8] There are minor typos in the appendix prompts and examples: 'hesistant' should be 'hesitant' in Listing 1, and 'Saurday' appears in the example transcript in Listing 8.
  4. [Section 4.1] Please clarify whether the 378 pointwise instances are exactly the two audio realizations of each of the 189 items, and whether all judges evaluate the same set of pairwise items; the n values in Table 7 suggest otherwise.
  5. [Figure 5] Please state more precisely how the GeminiGen vs GPTGen substitution was performed (e.g., whether only the annotation-generation LLM was swapped, or also the TTS rendering), and why the figure reports only the positional-emotion results.
  6. [Appendix F / References] The appendix prompt invokes Winoground and Winograd Schema; a citation for Winoground (e.g., Thrush et al., 2022) is missing from the reference list.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical comparisons of independently probed behaviors, not derivations from fitted parameters or self-citation chains.

full rationale

The paper's central claims are empirical, not derived from fitted parameters or self-citation constraints. The POINTWISE-to-PAIRWISE gap is a measured difference between two protocols on the same counterfactual items; neither accuracy is fit to the other, and the headline statement in Section 5.1 ('A model may distinguish the two counterfactuals when both are shown, but fail when the same audio cue must control a single-context decision') is an interpretation of that measured gap, not a reduction. The (P, O, J) decomposition in Section 3.3 is defined from task outputs and probe outcomes, not from a fitted parameter; the 'Potemkin' state pi_110 is a conjunction of independently probed behaviors, not an input to the audit. Construction uses Gemini for response generation, but the paper explicitly controls for generator-family bias with GPT-substituted annotations (Fig. 5), for quality balance with independent LLM/DNSMOS checks (Tab. 4), and for synthetic-speech artifacts with real-speech comparisons (Fig. 6). These controls may be imperfect, but they are not circular equations. The self-citations (Chandra et al. 2026; Huang et al. 2026) are background motivation and a human-disagreement aside; they are not invoked to forbid alternatives or to supply a uniqueness theorem. The human-validation limitation (single-turn accuracy 62-100%, Appendix E) is openly disclosed and affects interpretability, not circularity. No equation in the paper equates a predicted quantity to a fitted input, and no load-bearing premise is justified solely by a self-citation. The derivation chain is therefore self-contained for the claims it makes.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No free numerical constants are fitted to make a derivation work; the parameter-like choices are dataset-construction thresholds. The audit rests on domain assumptions about counterfactual validity, TTS fidelity, and probe composition, all acknowledged in the text. No new entities are postulated.

free parameters (2)
  • single-turn emotion-pair filtering ("obviously contrasting" pairs) = 189 items; 4 emotion-pair types
    Appendix B.1.2 filters CAVA EmoCF to "obviously contrasting" emotion pairs; the cutoff is a hand choice that determines the single-turn dataset composition and hence all reported accuracies.
  • Audio Scorer acceptance thresholds = 3 accepted attempts / 10 total attempts
    Appendix B.2.3 stops rendering after 3 accepted or 10 total attempts; this stopping rule filters which positional items survive and therefore affects the benchmark.
assumptions (4)
  • domain assumption Holding the transcript fixed while changing only the TTS rendering isolates paralinguistic evidence, so any change in judge decisions is attributable to the audio cue.
    Stated in Section 3.1 as the construct-validity premise of the audit.
  • domain assumption The counterfactual construction determines the correct response labels, independent of human majority vote.
    Section 4.3 says labels are determined by construction and human validation is only an internal-validity check.
  • domain assumption TTS renderings faithfully realize the intended emotions.
    Section 4.2 uses emotion2vec and acoustic heuristics for acceptance; these are imperfect proxies.
  • domain assumption Separately measured P, O, J probes on the same item compose into a valid failure attribution.
    Section 3.3 assumes probe scores per item can be combined into diagnostic states without considering order effects or prompt interaction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation." pith.science (2026). https://pith.science/paper/LJTPDE2X

@misc{pith2026260806718,
  author       = {Pith},
  title        = {Pith review of: Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LJTPDE2X}},
  note         = {Machine review of arXiv:2608.06718}
}
read the original abstract

Audio-language models (ALMs) are increasingly used as judges for speech-to-speech systems, but a judge that receives audio may not actually use paralinguistic evidence. We introduce counterfactual audits for paralinguistic response evaluation. Each audit item holds the transcript fixed while varying affect, prosody, or the timing of an affective shift, forcing a valid judge to track the audio cue rather than lexical content or response style. We evaluate ALM judges using a native one-context judgment protocol and a contrastive recoverability control, then further decompose each item into its constituent perception and response-mapping skills. This yields useful diagnostic states that identify different sources of judge failures. Across Gemini, GPT, and open audio models, we find that contrastive success often overstates native judge reliability, and that similar aggregate accuracies can hide different failure modes. These results suggest that ALM judges should not be evaluated by accuracy alone, instead requiring thorough behavioral audits before deployment.

Figures

Figures reproduced from arXiv: 2608.06718 by the authors.

Figure 1
Figure 1. Counterfactual audio-judge audit. Left: The audit holds the user’s words fixed while changing only the paralinguistic realization. Middle: POINTWISE provides one audio context and two candidate responses. If this fails, PAIRWISE provides both counterfactual audio contexts and responses. Component probes then test the constituent task skills to identify failure sources. Right: The judge correctly identifies the paral… view at source ↗
Figure 2
Figure 2. Task families in the audit. SINGLE-TURN-EMOTIONS isolates static paralinguistic response selection. EMOTIONAL-CONVERSATIONS is an earlier multi-turn prototype reported in the appendix. POSITIONAL-EMOTION has the timing and cause of the user’s affective shift determine the correct response. Statistic single-turn emo-conv pos-emo # Instances 378 500 1000 # Emotions 4 8 7 Avg turns 1.0 5.8 9.9 Avg sec/utterance 5.5 5.2… view at source ↗
Figure 3
Figure 3. Aggregated instrument-state distributions. Colors group the (P, O, J) states into reliable integrated judgment, component bottlenecks, Potemkin failures, shortcut-like successes, and full-stack failures [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Failure heatmaps. We quantify additional failure modes with conditional Potemkin rates, protocol collapse, and cue dependence. the same audio cue must control a single-context decision. We quantify this protocol collapse in more detail in [PITH_FULL_IMAGE:figures/full…
Figure 5
Figure 5. Figure 5: Generator bias. Accuracy on POSITIONAL￾EMOTION comparing GeminiGen and GPTGen gener￾ation sources. The superior performance of Gemini models is due to their genuine capability, and not a bias due to involvement in the construction pipeline. 5.3 Validity and robustness …
Figure 6
Figure 6. Figure 6: Generalization to real speech. Comparison of πpoj state proportions for synthetic vs real human speech from CAVA. The relationship has near perfect Pearson correlation and a slope close to 1 across all models. quality model (Reddy et al., 2021), confirming that irrelev…
Figure 7
Figure 7. Figure 7: Human validation interface for single-turn examples. Annotators listened to one emotional rendering of the fixed transcript and selected the response better matched to that audio. The interface mirrors the native POINTWISE model-judging protocol [PITH_FULL_IMAGE:figur…
Figure 8
Figure 8. Figure 8: Human validation interface for positional multi-turn examples. Annotators listened to the dialogue context and selected the response better matched to the user’s paralinguistic state. These examples require the listener to use the timing and cause of the affective shif…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 7 linked inside Pith

  1. [1]

    Journal of Artificial Intelligence Research , volume=

    Learning from Disagreement: A Survey , author=. Journal of Artificial Intelligence Research , volume=

  2. [2]

    We Need to Consider Disagreement in Evaluation

    Basile, Valerio and Fell, Michael and Fornaciari, Tommaso and Hovy, Dirk and Paun, Silviu and Plank, Barbara and Poesio, Massimo and Uma, Alexandra. We Need to Consider Disagreement in Evaluation. Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future. 2021. doi:10.18653/v1/2021.bppf-1.3

    doi:10.18653/v1/2021.bppf-1.3 2021

  3. [3]

    Transactions of the Association for Computational Linguistics , volume=

    Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations , author=. Transactions of the Association for Computational Linguistics , volume=

  4. [4]

    2021 , eprint=

    DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors , author=. 2021 , eprint=

    2021

  5. [5]

    2026 , eprint=

    DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality , author=. 2026 , eprint=

    2026

  6. [6]

    SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations

    Kottur, Satwik and Moon, Seungwhan and Geramifard, Alborz and Damavandi, Babak. SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. doi:10.18653/v1/2021.emnlp-main.401

    doi:10.18653/v1/2021.emnlp-main.401 2021

  7. [7]

    What do others think?

    "What do others think?": Task-Oriented Conversational Modeling with Subjective Knowledge , author=. 2023 , eprint=

    2023

  8. [8]

    2020 , eprint=

    MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling , author=. 2020 , eprint=

    2020

Show all 53 references
  1. [9]

    2025 , howpublished =

    CAVA: Comprehensive Assessment of Voice Assistants , author =. 2025 , howpublished =

    2025

  2. [10]

    N orm B ank: A Knowledge Bank of Situational Social Norms

    Ziems, Caleb and Dwivedi-Yu, Jane and Wang, Yi-Chia and Halevy, Alon and Yang, Diyi. N orm B ank: A Knowledge Bank of Situational Social Norms. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. doi:10.18653/v...

    2023 doi

  3. [11]

    The Body as a Medium of Expression , editor =

    Crystal, David , title =. The Body as a Medium of Expression , editor =. 1975 , pages =

    1975

  4. [12]

    The Thirteenth International Conference on Learning Representations , year=

    Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics , author=. The Thirteenth International Conference on Learning Representations , year=

  5. [13]

    2025 , url =

    Amazon Artificial General Intelligence , title =. 2025 , url =

    2025

  6. [14]

    2025 , eprint=

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author=. 2025 , eprint=

    2025

  7. [15]

    2025 , eprint=

    Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs , author=. 2025 , eprint=

    2025

  8. [16]

    2026 , eprint=

    DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment , author=. 2026 , eprint=

    2026

  9. [17]

    2025 , eprint=

    Qwen2.5-Omni Technical Report , author=. 2025 , eprint=

    2025

  10. [18]

    2025 , eprint=

    Voxtral , author=. 2025 , eprint=

    2025

  11. [19]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , journal=. Judging

  12. [20]

    The Fourteenth International Conference on Learning Representations , year=

    Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models , author=. The Fourteenth International Conference on Learning Representations , year=

  13. [21]

    2026 , eprint=

    ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction , author=. 2026 , eprint=

    2026

  14. [22]

    Do Audio LLM s Really LISTEN , or Just Transcribe? Measuring Lexical vs

    Chen, Jingyi and Guo, Zhimeng and Chun, Jiyun and Wang, Pichao and Perrault, Andrew and Elsner, Micha. Do Audio LLM s Really LISTEN , or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance. Proceedings of the 19th Conference of the E uropean Chapter of the A ...

    2026 doi

  15. [23]

    Proceedings of the AAAI Conference on Artificial Intelligence , author=

    SageLM: A Multi-aspect and Explainable Large Language Model for Speech Judgement , volume=. Proceedings of the AAAI Conference on Artificial Intelligence , author=. 2026 , month=. doi:10.1609/aaai.v40i36.40338 , number=

    2026 doi

  16. [24]

    2023 , note=

    Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang , booktitle=. 2023 , note=

    2023

  17. [25]

    Length-Controlled

    Dubois, Yann and Galambosi, Bal. Length-Controlled. arXiv preprint arXiv:2404.04475 , year=

    arXiv

  18. [26]

    2025 , eprint=

    MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocols , author=. 2025 , eprint=

    2025

  19. [27]

    Proceedings of the 42nd International Conference on Machine Learning , year=

    Potemkin Understanding in Large Language Models , author=. Proceedings of the 42nd International Conference on Machine Learning , year=

  20. [28]

    arXiv preprint arXiv:2503.05085 , year=

    S2S-Arena: Evaluating Speech2Speech Protocols on Instruction Following with Paralinguistic Information , author=. arXiv preprint arXiv:2503.05085 , year=

    arXiv

  21. [29]

    2021 , howpublished=

    2021

  22. [30]

    arXiv preprint arXiv:1904.08352 , year=

    MOSNet: Deep Learning based Objective Assessment for Voice Conversion , author=. arXiv preprint arXiv:1904.08352 , year=

    1904 arXiv

  23. [31]

    2003 , howpublished=

    2003

  24. [32]

    Yang, Shu-wen and Chi, Po-Han and Chuang, Yung-Sung and Lai, Cheng-I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan-Ting and others , booktitle=

  25. [33]

    Ribeiro, Marco Tulio and Wu, Tongshuang and Guestrin, Carlos and Singh, Sameer , booktitle=

  26. [34]

    arXiv preprint arXiv:2404.03606 , year=

    Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech , author=. arXiv preprint arXiv:2404.03606 , year=

    arXiv

  27. [35]

    arXiv preprint arXiv:2302.04174 , year=

    Multimodal Chain-of-Thought Reasoning in Language Models , author=. arXiv preprint arXiv:2302.04174 , year=

    arXiv

  28. [36]

    2025 , address=

    Lee, Yukyung and Kim, JoongHoon and Kim, Jaehee and Cho, Hyowon and Kang, Jaewook and Kang, Pilsung and Kim, Najoung , booktitle=. 2025 , address=

    2025

  29. [37]

    2025 , address=

    Li, Minzhi and Liu, Zhengyuan and Deng, Shumin and Joty, Shafiq and Chen, Nancy and Kan, Min-Yen , booktitle=. 2025 , address=

    2025

  30. [38]

    arXiv preprint arXiv:2507.12705 , year=

    AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation , author=. arXiv preprint arXiv:2507.12705 , year=

    arXiv

  31. [39]

    arXiv preprint arXiv:2506.05984 , year=

    Audio-Aware Large Language Models as Judges for Speaking Styles , author=. arXiv preprint arXiv:2506.05984 , year=

    arXiv

  32. [40]

    arXiv preprint arXiv:2505.09558 , year=

    WavReward: Spoken Dialogue Models With Generalist Reward Evaluators , author=. arXiv preprint arXiv:2505.09558 , year=

  33. [41]

    ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year=

    Task Oriented Dialogue as a Catalysis for Self-Supervised Automatic Speech Recognition , author=. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year=

    2024

  34. [42]

    emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation , author=. Proc. ACL 2024 Findings , year=

    2024

  35. [43]

    Key-Value Retrieval Networks for Task-Oriented Dialogue

    Eric, Mihail and Krishnan, Lakshmi and Charette, Francois and Manning, Christopher D. Key-Value Retrieval Networks for Task-Oriented Dialogue. Proceedings of the 18th Annual SIG dial Meeting on Discourse and Dialogue. 2017. doi:10.18653/v1/W17-5506

    2017 doi

  36. [44]

    M ulti WOZ - A Large-Scale Multi-Domain W izard-of- O z Dataset for Task-Oriented Dialogue Modelling

    Budzianowski, Pawe and Wen, Tsung-Hsien and Tseng, Bo-Hsiang and Casanueva, I \ n igo and Ultes, Stefan and Ramadan, Osman and Ga s i \'c , Milica. M ulti WOZ - A Large-Scale Multi-Domain W izard-of- O z Dataset for Task-Oriented Dialogue Modelling. Proceedings of the 2018 Con...

    2018 doi

  37. [45]

    and Lastras, Luis and Lasecki, Walter S

    Gunasekara, Chulaka and Kummerfeld, Jonathan K. and Lastras, Luis and Lasecki, Walter S. , title =. AAAI: Workshop on Dialog System Tech Challenges , year =

  38. [46]

    IEEE Transactions on Affective Computing , volume =

    CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset , author =. IEEE Transactions on Affective Computing , volume =. 2014 , doi =

    2014

  39. [47]

    PLOS ONE , volume =

    The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English , author =. PLOS ONE , volume =. 2018 , doi =

    2018

  40. [48]

    Hearing Between the Lines: Unlocking the Reasoning Power of LLM s for Speech Evaluation

    Chandra, Arjun and Miller, Kevin and Ravichandran, Venkatesh and Papayiannis, Constantinos and Saligrama, Venkatesh. Hearing Between the Lines: Unlocking the Reasoning Power of LLM s for Speech Evaluation. Findings of the A ssociation for C omputational L inguistics: EACL 2026...

    2026 doi

  41. [49]

    2014 , publisher =

    Standards for Educational and Psychological Testing , author =. 2014 , publisher =

    2014

  42. [50]

    Psychological Review , volume =

    The Concept of Validity , author =. Psychological Review , volume =. 2004 , doi =

    2004

  43. [51]

    Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages =

    Measurement and Fairness , author =. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages =. 2021 , publisher =

    2021

  44. [52]

    Speech Communication , volume =

    Vocal Communication of Emotion: A Review of Research Paradigms , author =. Speech Communication , volume =. 2003 , doi =

    2003

  45. [53]

    2019 , publisher =

    Poria, Soujanya and Hazarika, Devamanyu and Majumder, Navonil and Naik, Gautam and Cambria, Erik and Mihalcea, Rada , booktitle =. 2019 , publisher =

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.