The reviewed record of science sign in
Pith

arxiv: 2607.05365 · v1 · pith:V3Y7IO74 · submitted 2026-07-06 · cs.CL · cs.AI· eess.AS

SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models

Thomas Thebaud , Yuzhe Wang , Hao Zhang , Sathvik Manikantan Napa Ugandhar , Ashish Hallur , Georgi Tinchev , Venkatesh Ravichandran , Laureano Moro-Velazquez This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-07 14:37 UTCglm-5.2pith:V3Y7IO74record.jsonopen to challenge →

classification cs.CL cs.AIeess.AS
keywords modelslanguagebenchmarkdialectinterpersonalnaturalnessqualityspearbench
0
0 comments X

The pith

Clean speech isn't natural speech: S2S models ace audio, fail conversation

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces SPEARBench, a benchmark that evaluates streaming speech-to-speech (S2S) language models not just on audio quality or intelligibility, but on whether they actually behave like human conversational partners. The authors extract question-answer dialogue snippets from the Seamless Interaction corpus, feed the context and question audio into seven S2S models, and compare each model's spoken answer against the original human answer across a wide set of dimensions: response latency, interruptions, turn-taking naturalness, dialect consistency and entrainment, emotional naturalness, interpersonal stance, and prosodic variation. The central finding is that current S2S models produce speech that is often cleaner and more intelligible than human speech (higher MOS scores, lower WER), yet they consistently diverge from human conversational behavior in ways that standard benchmarks miss. Models respond too slowly (959-2724ms vs. 742ms for humans), show near-zero dialectal entrainment to the questioner's accent, flatten pitch variation (0.118-0.321 vs. human 0.321), and fail to emotionally entrain on arousal or dominance even when they roughly match emotional valence. The paper's core claim is that signal-level quality and conversational naturalness are distinct properties, and that existing evaluation protocols systematically miss the gap because they do not measure the interactional and paralinguistic dimensions that make spoken dialogue feel human.

Core claim

The paper's central discovery is a dissociation between acoustic quality and conversational naturalness in S2S language models. Models can score higher than humans on speech quality (UTMOS 3.72-4.36 vs. human 2.22) and lower on word error rate (5.8-26.7% vs. human 27.1%), while simultaneously exhibiting substantially worse conversational behavior: latencies 1.3-3.7 times longer than humans, near-zero dialectal entrainment (beta 0.28-0.54 vs. human 0.53), reduced pitch variation, and weak emotional entrainment on arousal and dominance dimensions. This gap is invisible to benchmarks that focus on utterance-level correctness or audio fidelity alone.

What carries the argument

SPEARBench operates as a three-stage pipeline: (1) data preparation extracts two-speaker question-answer dialogues from the Seamless Interaction corpus by selecting turns where the second-to-last turn ends with a question; (2) LLM inference feeds context and question audio to each evaluated S2S model and records the generated waveform and start time; (3) evaluation applies a suite of automatic evaluator models including DualTurn for turn-taking naturalness, TRACE for emotional naturalness, Voxlect-English for dialect profiling, GPT-audio for interpersonal stance, UTMOS for speech quality, Whisper/Qwen3-ASR for intelligibility, and Silero VAD for latency and interruption detection. Human原始答案s

If this is right

  • S2S model developers may need to optimize for conversational timing, prosodic expressivity, and dialectal adaptation as first-class objectives rather than treating them as downstream effects of improved speech quality.
  • The finding that models with high UTMOS and low WER still feel unnatural suggests that leaderboards ranked on audio fidelity alone are misleading for real-time conversational applications.
  • Dialectal entrainment (beta near zero for most models) implies that S2S systems may be failing to build rapport with speakers who have non-standard accents, a fairness and inclusion concern that current benchmarks do not flag.
  • The reduced pitch variation across all models relative to humans points to a systematic prosodic flattening that may stem from how speech tokenizers or decoding pipelines constrain expressivity, suggesting a target for architectural improvement.
  • If the benchmark's automatic evaluators correlate with human perceptual judgments (which the paper has not yet verified), SPEARBench could become a cheap proxy for large-scale human naturalness studies, reducing the cost of iterative S2S development.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The dissociation between audio quality and conversational naturalness may partially explain user dissatisfaction with voice assistants that test well on intelligibility but feel robotic in practice; the benchmark provides a vocabulary for naming what is missing.
  • The near-zero dialectal entrainment could be a deliberate consequence of voice-cloning or decoding pipelines that lock to a single accent profile rather than adapting to interlocutor cues; if so, enabling accent-conditional generation could be a direct architectural fix.
  • The finding that full-duplex models show higher latency than half-duplex models is counterintuitive and may reflect that current full-duplex implementations spend computational resources on overlap detection rather than on fast response initiation, suggesting an engineering tradeoff that has not been optimized.
  • The pitch-variation reduction, if replicated with human perceptual validation, could serve as a simple diagnostic signal: a single scalar (normalized pitch std) might flag models that will feel emotionally flat before full conversational evaluation is run.

Load-bearing premise

The benchmark assumes that its automatic evaluator models (for emotional naturalness, turn-taking, dialect, and stance) produce scores that correspond to what humans would actually perceive as natural, but this assumption has not yet been validated against human perceptual judgments.

What would settle it

If the automatic evaluator models systematically assign high naturalness scores to acoustic patterns that humans do not actually perceive as natural, then the gap between model and human conversational behavior reported by SPEARBench could be an artifact of the evaluators rather than a real property of the S2S models.

Figures

Figures reproduced from arXiv: 2607.05365 by Ashish Hallur, Georgi Tinchev, Hao Zhang, Laureano Moro-Velazquez, Sathvik Manikantan Napa Ugandhar, Thomas Thebaud, Venkatesh Ravichandran, Yuzhe Wang.

Figure 1
Figure 1. Figure 1: SPEARBench pipeline. C. Existing Benchmarks for Speech-to-Speech Language Mod￾els Evaluation Many benchmarks evaluate speech processing models and LLMs. Some focus on speech understanding tasks such as transcription, spoken question answering, speech translation, or audio reasoning, including SUPERB-style benchmarks for speech foundation models [40]. Others compare LLM abilities in text-only [41], [42] or … view at source ↗
Figure 4
Figure 4. Figure 4: Scatter plots of the Arousal/Valence/Dominance pre [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Dialect profiles. This pointplot shows the average [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distributions of the pitch features across all evaluated [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Interpersonal stance results. The pointplot reports the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
read the original abstract

Streaming speech-to-speech language models aim to answer spoken queries directly with synthetic speech. However, standard speech and text benchmarks do not capture whether these systems behave naturally in conversations, where timing, turn-taking, prosody, interpersonal stance, language and dialect consistency, and relationship-aware appropriateness jointly shape perceived quality. We introduce SPEARBench, a benchmark for evaluating naturalness in speech-to-speech language models from question-answer interactions. SPEARBench constructs controlled dialogue prompts from the Seamless Interaction corpus, runs inference across multiple models, and evaluates generated answers using a multidimensional protocol that covers response latency, interruptions, speech quality, ASR robustness, language and dialect consistency, emotional naturalness, interpersonal stance, and explainable distributional baselines. The benchmark includes original human answers as a reference condition and reports results for several contemporary models. Results show that current models can achieve high signal-level quality and low ASR error while still differing from human conversational behavior in latency, overlap, dialect preservation, emotional adaptation, and interpersonal stance dynamics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 9 minor

Summary. SPEARBench introduces a multidimensional benchmark for evaluating conversational naturalness in streaming speech-to-speech (S2S) language models. The benchmark extracts question-answer dialogue pairs from the Seamless Interaction corpus, runs inference across seven S2S models (including a human reference condition), and evaluates outputs along dimensions including latency, interruptions, speech quality, ASR robustness, language/dialect consistency, emotional naturalness, interpersonal stance, turn-taking naturalness, and prosodic features. The central finding is that current S2S models achieve high signal-level quality (UTMOS, WER) but diverge from human conversational behavior in timing, prosody, dialect entrainment, emotional adaptation, and stance dynamics. The evaluation pipeline integrates several prior tools from the authors' group (TRACE, StanceBench, TurnNat, Voxlect, prosodic baselines) into a unified open-source framework with a public website.

Significance. The paper addresses a genuine gap: existing S2S benchmarks focus on isolated utterances or single dimensions rather than multidimensional conversational naturalness. The release of curated evaluation data, open-source code, and a public reporting platform is a concrete contribution to reproducibility. The inclusion of a human reference condition alongside seven diverse S2S models provides useful comparative context. The finding that models can score well on signal quality while diverging from human conversational behavior on multiple dimensions is well-supported by the objective metrics (latency via Silero VAD, pitch variation via f0 extraction, answer duration) and is a valuable empirical observation for the field. However, the construct validity of the evaluator-dependent dimensions remains unverified, which limits the strength of conclusions drawn from those components.

major comments (3)
  1. §III-C.6 and Table II: GPT-audio is used as the interpersonal stance judge while GPT-audio-1.5 is simultaneously one of the seven evaluated S2S models. In Table III, GPT-audio-1.5 achieves the highest 'same stance' percentage (98.0%) among all models. This evaluator-contestant overlap is not acknowledged in the manuscript. The paper should either (a) use a different stance judge for the GPT-audio-1.5 condition, (b) report a sensitivity analysis with an alternative judge, or (c) explicitly disclose the overlap and discuss its potential impact. This is load-bearing for the stance dimension results, though the central claim about model-human behavioral gaps also rests on objective metrics independent of this conflict.
  2. §V and §III-C: The paper states that human validation of the platform is planned but not yet performed, and that each evaluator model was 'previously validated' in its original publication. However, all five evaluator components (TRACE [10], StanceBench [11], TurnNat [12], Voxlect [14], distributional baselines [15]) are from the authors' own prior work, and their original validations were on different data distributions than SPEARBench's Seamless Interaction extraction. The claim in §V that 'each evaluation component is based on a previously validated model or metric, providing independent evidence of the reliability of the benchmark's measured dimensions' overstates the evidence: validation on a different corpus does not establish construct validity for this specific benchmark context. The paper should temper this claim and clarify what 'validated' means in each case.
  3. §III-C.5, Table III: The emotional naturalness scores show humans at 11.10 and models ranging from 9.21 to 10.92, with the paper reporting p < 10^-13 for all models via paired Friedman test. However, the practical significance of these small absolute differences is unclear without knowing the score scale, the number of test items, or the effect size. The paper should report effect sizes or clarify the score range and its interpretation to support the claim that models 'differ from human conversational behavior' on this dimension.
minor comments (9)
  1. Table III is extremely dense and difficult to parse. Consider splitting into sub-tables by metric group (speech quality, interruptions, turn-taking, etc.) or using consistent column groupings with clearer visual separation.
  2. §III-A.2: The backchannel silence threshold of 0.5 seconds for defining turns is stated without justification. A brief citation or sensitivity note would strengthen the dialogue extraction protocol.
  3. §III-C.4: The merging of five dialect categories (Irish, Northern Irish, Scottish, English, Welsh) into 'British Isles' and the exclusion of 'Others' reduces the dialect profile from 16 to 11 dimensions. The rationale for this specific merging scheme should be briefly justified, as it affects entrainment measurements.
  4. §III-C.4: The dialectal entrainment metric is defined as the least squares regression coefficient between question and answer dialect profiles. It is unclear whether this is a per-dimension regression or a single scalar. The human reference shows β=0.53 while models range from 0.28 to 0.54; the interpretation of these values should be clarified.
  5. §IV-B: The paper states that most full-duplex models exhibit higher latency than half-duplex models, but Table III shows half-duplex Qwen2.5-Omni-7B at 1176ms and full-duplex Gemini-2.5 at 2137ms, while non-streaming GPT-audio-1.5 is at 959ms. The claim about the duplex-latency relationship is not clearly supported by the data as presented.
  6. Figure 2: The histograms for turn-taking and emotional naturalness scores are mentioned but the figure quality and labeling should be improved to show clearer separation between models and the human reference.
  7. §III-C.1: The paper uses two ASR systems (whisper-large-v3 and qwen3-ASR-0.6B) to reduce bias, but Table III reports a single WER and CER per model. It should be clarified whether these are averages across both ASR systems or from one system.
  8. The paper references several of its own prior works ([10], [11], [12], [14], [15]) as core evaluation components. While these are cited, the degree of self-reliance on prior tools for the evaluation pipeline should be more transparently discussed, including whether alternative external evaluators were considered.
  9. §III-B: The start time computation depends on model serving interfaces, which the paper acknowledges as a limitation in §V. The specific interfaces used for each model and their potential impact on latency measurements should be documented, as different serving backends may introduce variable delays unrelated to model behavior.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three major comments: (1) an evaluator-contestant overlap between GPT-audio as stance judge and GPT-audio-1.5 as an evaluated model, (2) an overstated claim about construct validity given that all evaluator components are from the authors' prior work and were validated on different data distributions, and (3) insufficient reporting of effect sizes and score scale for the emotional naturalness dimension. We agree with all three points and will revise the manuscript accordingly. The overlap in Comment 1 is a genuine oversight that we will address through disclosure and sensitivity analysis. The construct validity claim in Comment 2 is overstated and will be tempered. The effect size reporting in Comment 3 will be added. We note that the central finding—that models achieve high signal-level quality while diverging from human conversational behavior—also rests on objective metrics (latency via Silero VAD, pitch variation via f0 extraction, answer duration) that are independent of the contested evaluator components.

read point-by-point responses

  1. Referee: §III-C.6 and Table II: GPT-audio is used as the interpersonal stance judge while GPT-audio-1.5 is simultaneously one of the seven evaluated S2S models. In Table III, GPT-audio-1.5 achieves the highest 'same stance' percentage (98.0%) among all models. This evaluator-contestant overlap is not acknowledged in the manuscript.

    Authors: The referee is correct that this overlap exists and was not acknowledged in the manuscript. We will address this in the revision through a combination of options (b) and (c) from the referee's suggested remedies. Specifically, we will (1) explicitly disclose the overlap in §III-C.6, (2) add a discussion of its potential impact on the stance dimension results, and (3) conduct and report a sensitivity analysis using an alternative audio-language model as stance judge for the GPT-audio-1.5 condition. We note that GPT-audio was selected as the stance judge because it achieved the highest average AUROC (0.83) across all stance dimensions in the StanceBench evaluation, not because of any preference for the model family. Nevertheless, we agree that the overlap should have been disclosed and its impact assessed. We also note that the stance dimension is one of several evaluated dimensions, and the central claim about model-human behavioral gaps rests substantially on objective metrics (latency, pitch variation, answer duration, interruption rates) that are independent of this evaluator-contestant overlap. revision: yes

  2. Referee: §V and §III-C: The paper states that human validation of the platform is planned but not yet performed, and that each evaluator model was 'previously validated' in its original publication. However, all five evaluator components (TRACE [10], StanceBench [11], TurnNat [12], Voxlect [14], distributional baselines [15]) are from the authors' own prior work, and their original validations were on different data distributions than SPEARBench's Seamless Interaction extraction. The claim in §V that 'each evaluation component is based on a previously validated model or metric, providing independent evidence of the reliability of the benchmark's measured dimensions' overstates the evidence.

    Authors: We agree with the referee that the claim in §V overstates the evidence. Validation on a different corpus does not establish construct validity for SPEARBench's specific data distribution, and the fact that all five evaluator components originate from our group further weakens the independence claim. In the revision, we will (1) temper the statement in §V to accurately characterize what 'validated' means in each case—namely, that each component was evaluated on its original publication's data distribution, not on SPEARBench's extraction from Seamless Interaction, (2) clarify the specific validation context for each component (e.g., TRACE's 97.01% accuracy was on real and simulated datasets from its original study, StanceBench's 0.83 AUROC was on its own evaluation set, etc.), and (3) explicitly state that construct validity for SPEARBench's specific context remains unverified pending the planned human validation study. We will also acknowledge the shared-authorship concern as a limitation. revision: yes

  3. Referee: §III-C.5, Table III: The emotional naturalness scores show humans at 11.10 and models ranging from 9.21 to 10.92, with the paper reporting p < 10^-13 for all models via paired Friedman test. However, the practical significance of these small absolute differences is unclear without knowing the score scale, the number of test items, or the effect size.

    Authors: The referee is right that the practical significance of the emotional naturalness differences is unclear from the current reporting. The score range is not bounded on a fixed perceptual scale; as noted in §III-C.5, values should be interpreted relative to the human reference condition and across systems rather than as absolute perceptual scores. However, we did not make this sufficiently clear in the results section, nor did we report effect sizes or sample sizes. In the revision, we will (1) clarify the score scale and its interpretation in §IV-E, (2) report the number of test items used in the Friedman test, (3) report effect sizes (we will compute Kendall's W for the Friedman test and also report Cohen's d or rank-biserial correlation for pairwise model-vs-human comparisons), and (4) discuss whether the small absolute differences constitute practically meaningful deviations from human behavior or whether the more compelling evidence for model-human divergence on the emotional dimension comes from the entrainment correlations (Table III, Figure 4), where models show near-zero arousal and dominance correlations compared to the human reference's 0.53 and 0.51 respectively. revision: yes

Circularity Check

0 steps flagged

Heavy self-citation in the evaluation pipeline, but the central claim is also supported by external metrics; no step reduces to its inputs by construction.

full rationale

The paper's central claim—that S2S models achieve high signal quality but diverge from human conversational behavior—is supported by a mix of external and self-cited metrics. The external metrics (UTMOS [39] for speech quality, whisper-large-v3 [50] and qwen3-ASR [21] for WER, Silero VAD [51] for latency/interruptions, MMS [52] for language ID, Voxlect [14] for dialect, and direct f0 extraction for pitch variation) are independent of the authors' prior work and independently support the claim that models have longer latency (959–2724ms vs. 742ms human), lower pitch variation (0.118–0.321 vs. 0.321), and near-zero dialectal entrainment. The self-cited components (TRACE [10], StanceBench [11], TurnNat [12], prosody baselines [15]) are pre-trained evaluator models or methodologies applied to new data (the Seamless Interaction corpus), not fits to the data being evaluated. No equation or metric is defined in terms of the quantity it claims to predict. The GPT-audio evaluator-contestant overlap (GPT-audio [27] serves as stance judge while GPT-audio-1.5 is an evaluated model) is a genuine evaluator-bias concern, but it is a correctness risk, not a circularity: the stance metric is not defined in terms of GPT-audio's self-evaluation in a way that would make the result tautological. The self-citations are load-bearing for the emotional naturalness, turn-taking, and stance dimensions specifically, but the paper's headline finding rests substantially on objective, externally-sourced measurements. Score 2 reflects the heavy self-citation pattern without a construction-level reduction.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical entities, particles, or mathematical objects. The free parameters are design choices in the benchmark pipeline (turn-segmentation threshold, dialect category merging). The axioms are domain assumptions about evaluator validity and data representativeness, plus one ad hoc assumption about judge impartiality. No invented entities are postulated.

free parameters (2)
  • Backchannel silence threshold = 0.5s
    Set in §III-A to define a turn boundary: 'a succession of utterances from a given speaker without interruption of backchannel over 0.5 seconds.' This threshold determines how turns are segmented and thus which dialogues are selected.
  • Dialect category merging = 11 categories (from 16)
    §III-C.4: Irish, Northern Irish, Scottish, English, and Welsh are merged into 'British Isles'; 'Others' is ignored. This ad hoc grouping affects the dialectal variance and entrainment metrics.
axioms (4)
  • domain assumption Automatic evaluator models (TRACE, DualTurn, Voxlect, GPT-audio judge) produce scores that correlate with human perceptual judgments of naturalness.
    Invoked throughout §III-C and §IV. The benchmark's construct validity depends on this assumption, which the paper acknowledges is unvalidated (§V: 'validating the platform with human judgments' is future work).
  • domain assumption Question-answer dialogues extracted from Seamless Interaction are representative of natural conversational behavior.
    Invoked in §III-A. The benchmark selects dialogues where the second-to-last turn ends with '?', which may bias toward information-seeking interactions and away from other conversational modes.
  • domain assumption Latency measurements from model serving interfaces reflect the latency a user would experience in deployment.
    Invoked in §III-B. The paper acknowledges (§V) that 'timing measures depend partly on model serving interfaces,' which may not generalize to production deployments.
  • ad hoc to paper GPT-audio can serve as an impartial stance judge even when evaluating GPT-audio-1.5 as a contestant.
    Invoked in §III-C.6. The paper selects GPT-audio as the stance judge based on StanceBench AUROC results [11] but does not address the conflict of interest when the judge model family overlaps with a contestant.

pith-pipeline@v1.1.0-glm · 16545 in / 3244 out tokens · 209778 ms · 2026-07-07T14:37:14.857572+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 55 canonical work pages · 22 internal anchors

  1. [1]

    Large language models: a survey of their development, capabilities, and applications,

    Y . Annepaka and P. Pakray, “Large language models: a survey of their development, capabilities, and applications,”Knowledge and Informa- tion Systems, vol. 67, no. 3, pp. 2967–3022, 2025

    work page 2025

  2. [2]

    On The Landscape of Spoken Language Models: A Comprehensive Survey

    S. Arora, K.-W. Chang, C.-M. Chien, Y . Peng, H. Wu, Y . Adi, E. Dupoux, H.-Y . Lee, K. Livescu, and S. Watanabe, “On the landscape of spoken language models: A comprehensive survey,”arXiv preprint arXiv:2504.08528, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  3. [3]

    A full-duplex speech dialogue scheme based on large language model,

    P. Wang, S. Lu, Y . Tang, S. Yan, W. Xia, and Y . Xiong, “A full-duplex speech dialogue scheme based on large language model,”Advances in Neural Information Processing Systems, vol. 37, pp. 13 372–13 403, 2024

    work page 2024

  4. [4]

    Human conversational behavior,

    R. I. Dunbar, A. Marriott, and N. D. Duncan, “Human conversational behavior,”Human nature, vol. 8, no. 3, pp. 231–246, 1997

    work page 1997

  5. [5]

    6 conversation analytic approaches to the relevance and uses of relationship categories in interaction,

    A. Pomerantz and J. Mandelbaum, “6 conversation analytic approaches to the relevance and uses of relationship categories in interaction,” in Handbook of language and social interaction. Psychology Press, 2004, pp. 149–171

    work page 2004

  6. [6]

    τ-voice: Benchmarking full-duplex voice agents on real-world domains,

    S. Ray, K. Dhandhania, V . Barres, and K. Narasimhan, “τ-voice: Benchmarking full-duplex voice agents on real-world domains,”arXiv preprint arXiv:2603.13686, 2026

    work page arXiv 2026

  7. [7]

    Wildspeech-bench: Benchmarking end-to-end speechllms in the wild,

    L. Zhang, J. Zhang, B. Lei, C. Wu, A. Liu, W. Jia, and X. Zhou, “Wildspeech-bench: Benchmarking end-to-end speechllms in the wild,” arXiv preprint arXiv:2506.21875, 2025

    work page arXiv 2025

  8. [8]

    Full-duplex-bench v1. 5: Evaluating overlap handling for full-duplex speech models,

    G.-T. Lin, S.-Y . S. Kuan, Q. Wang, J. Lian, T. Li, S. Watanabe, and H.-y. Lee, “Full-duplex-bench v1. 5: Evaluating overlap handling for full-duplex speech models,” inICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2026, pp. 19 447–19 451

    work page 2026

  9. [9]

    Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset

    V . Agrawal, A. Akinyemi, K. Alvero, M. Behrooz, J. Buffalini, F. M. Carlucci, J. Chen, J. Chen, Z. Chen, S. Chenget al., “Seamless inter- action: Dyadic audiovisual motion modeling and large-scale dataset,” arXiv preprint arXiv:2506.22554, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    TRACE: Temporal Relationship-Aware Conversational Entrainment Detection in Dyadic Speech

    S. M. N. Ugandhar, H. Zhang, A. Gunzler, Y . Wang, T. Thebaud, G. Tinchev, V . Ravichandran, and L. Moro-Vel´azquez, “Trace: Temporal relationship-aware conversational entrainment detection in dyadic speech,” 2026. [Online]. Available: https://arxiv.org/abs/2606.30543

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  11. [11]

    The embedded deformation problem for monomial ideals

    Y . Wang, T. Thebaud, J. Hu, J. Villalba-Lopez, V . Ravichandran, G. Tinchev, N. Dehak, and L. Moro-Vel´azquez, “Stancebench: A bench- mark for audio llm-based interpersonal stance evaluation from speech,” arXiv preprint arXiv:2506.10827, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  12. [12]

    TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue

    H. Zhang, T. Thebaud, G. Tinchev, V . Ravichandran, and L. Moro- Velazquez, “Turnnat: Automatic evaluation of turn-taking naturalness in dyadic spoken dialogue,” 2026. [Online]. Available: https://arxiv.org/ abs/2607.01345

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  13. [13]

    Vox-Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker and Speech Traits

    T. Feng, J. Lee, A. Xu, Y . Lee, T. Lertpetchpun, X. Shi, H. Wang, T. Thebaud, L. Moro-Velazquez, D. Byrdet al., “V ox-profile: A speech foundation model benchmark for characterizing diverse speaker and speech traits,”arXiv preprint arXiv:2505.14648, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    V oxlect: A speech foundation model benchmark for modeling dialects and regional languages around the globe,

    T. Feng, K. Huang, A. Xu, X. Shi, T. Lertpetchpun, J. Lee, Y . Lee, D. Byrd, and S. Narayanan, “V oxlect: A speech foundation model benchmark for modeling dialects and regional languages around the globe,” inProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V . 1, 2026, pp. 2640–2651

    work page 2026

  15. [15]

    Reference-based prosody and rhythm evaluation for spoken dialogue systems,

    A. Hallur, T. Thebaud, G. Tinchev, V . Ravichandran, and L. Moro- Velazquez, “Reference-based prosody and rhythm evaluation for spoken dialogue systems,” 2026. [Online]. Available: https://arxiv.org/abs/2606. 31055

    work page 2026

  16. [16]

    Gpt-4 technical report,

    OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmadet al., “Gpt-4 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2303. 08774

    work page 2024

  17. [17]

    Qwen3-Omni Technical Report

    J. Xu, Z. Guo, H. Hu, Y . Chu, X. Wang, J. He, Y . Wang, X. Shi, T. He, X. Zhuet al., “Qwen3-omni technical report,”arXiv preprint arXiv:2509.17765, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    Gemini: A Family of Highly Capable Multimodal Models

    G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millicanet al., “Gemini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [19]

    SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtokenizer: Unified speech tokenizer for speech large language models,”arXiv preprint arXiv:2308.16692, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  20. [20]

    Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers

    Y . Gong, S. Khurana, L. Karlinsky, and J. Glass, “Whisper-at: Noise- robust automatic speech recognizers are also strong general audio event taggers,”arXiv preprint arXiv:2307.03183, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  21. [21]

    Qwen3-ASR Technical Report

    X. Shi, X. Wang, Z. Guo, Y . Wang, P. Zhang, X. Zhang, Z. Guo, H. Hao, Y . Xi, B. Yanget al., “Qwen3-asr technical report,”arXiv preprint arXiv:2601.21337, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  22. [22]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    work page 2021

  23. [23]

    wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

    A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” CoRR, vol. abs/2006.11477, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  24. [24]

    Wavlm: Large-scale self-supervised pre- training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self-supervised pre- training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    work page 2022

  25. [25]

    WavLLM: Towards Robust and Adaptive Speech Large Language Model

    S. Hu, L. Zhou, S. Liu, S. Chen, H. Hao, J. Pan, X. Liu, J. Li, S. Sivasankaran, L. Liuet al., “Wavllm: Towards robust and adaptive speech large language model,”arXiv preprint arXiv:2404.00656, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  26. [26]

    Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  27. [27]

    gpt-audio-1.5,

    OpenAI, “gpt-audio-1.5,” Large language model, 2026. [Online]. Available: https://developers.openai.com/api/docs/models/gpt-audio-1.5

    work page 2026

  28. [28]

    Qwen2.5-Omni Technical Report

    J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y . Fan, K. Dang, B. Zhang, X. Wang, Y . Chu, and J. Lin, “Qwen2.5-omni technical report,”arXiv preprint arXiv:2503.20215, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  29. [29]

    gpt-realtime-2,

    OpenAI, “gpt-realtime-2,” Large language model, 2026. [Online]. Available: https://developers.openai.com/api/docs/models/gpt-realtime-2

    work page 2026

  30. [30]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosenet al., “Gem- ini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,”arXiv preprint arXiv:2507.06261, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  31. [31]

    gemini-3.1-flash-live-preview,

    Google AI, “gemini-3.1-flash-live-preview,” Large language model,

    work page

  32. [32]

    Available: https://ai.google.dev/gemini-api/docs/models/ gemini-3.1-flash-live-preview

    [Online]. Available: https://ai.google.dev/gemini-api/docs/models/ gemini-3.1-flash-live-preview

    work page

  33. [33]

    Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming

    Z. Xie and C. Wu, “Mini-omni: Language models can hear, talk while thinking in streaming,”arXiv preprint arXiv:2408.16725, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  34. [34]

    Couper-Kuhlen and M

    E. Couper-Kuhlen and M. Selting,Prosody in conversation: Interac- tional studies. Cambridge University Press, 1996, no. 12

    work page 1996

  35. [35]

    Phonetics and prosody in conversation,

    G. Walker, “Phonetics and prosody in conversation,”The handbook of conversation analysis, pp. 455–474, 2012

    work page 2012

  36. [36]

    Classifying conversational entrainment of speech behavior: An expanded framework and review,

    C. J. Wynn and S. A. Borrie, “Classifying conversational entrainment of speech behavior: An expanded framework and review,”Journal of Phonetics, vol. 94, p. 101173, 2022

    work page 2022

  37. [37]

    A cross-linguistic analysis of the temporal dynamics of turn-taking cues using machine learning as a descriptive tool,

    P. Brusco, J. Vidal, ˇS. Be ˇnuˇs, and A. Gravano, “A cross-linguistic analysis of the temporal dynamics of turn-taking cues using machine learning as a descriptive tool,”Speech Communication, vol. 125, pp. 24–40, 2020

    work page 2020

  38. [38]

    Real-time changes to social dynamics in human-robot turn-taking,

    J. S. Smith, C. Chao, and A. L. Thomaz, “Real-time changes to social dynamics in human-robot turn-taking,” in2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2015, pp. 3024–3029

    work page 2015

  39. [39]

    Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics

    S. Arora, Z. Lu, C.-C. Chiu, R. Pang, and S. Watanabe, “Talking turns: Benchmarking audio foundation models on turn-taking dynamics,”arXiv preprint arXiv:2503.01174, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  40. [40]

    UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “Utmos: Utokyo-sarulab system for voicemos challenge 2022,”arXiv preprint arXiv:2204.02152, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  41. [41]

    SUPERB: Speech processing Universal PERformance Benchmark

    S.-w. Yang, P.-H. Chi, Y .-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y . Y . Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Linet al., “Superb: Speech processing universal performance benchmark,”arXiv preprint arXiv:2105.01051, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  42. [42]

    Wildbench: Benchmarking llms with chal- lenging tasks from real users in the wild,

    B. Y . Lin, Y . Deng, K. Chandu, A. Ravichander, V . Pyatkin, N. Dziri, R. Le Bras, and Y . Choi, “Wildbench: Benchmarking llms with chal- lenging tasks from real users in the wild,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 47 852–47 870

    work page 2025

  43. [43]

    Mt-bench-101: A fine-grained benchmark for evaluat- ing large language models in multi-turn dialogues,

    G. Bai, J. Liu, X. Bu, Y . He, J. Liu, Z. Zhou, Z. Lin, W. Su, T. Ge, B. Zhenget al., “Mt-bench-101: A fine-grained benchmark for evaluat- ing large language models in multi-turn dialogues,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 7421–7454

    work page 2024

  44. [44]

    Air-bench: Benchmarking large audio-language models via generative comprehension,

    Q. Yang, J. Xu, W. Liu, Y . Chu, Z. Jiang, X. Zhou, Y . Leng, Y . Lv, Z. Zhao, C. Zhouet al., “Air-bench: Benchmarking large audio-language models via generative comprehension,” inProceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 1979–1998

    work page 2024

  45. [45]

    GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models

    R. Tan and W. Zhang, “Globeaudio: A multilingual multicultural bench- mark for naturalistic evaluation of large audio-language models,”arXiv preprint arXiv:2606.08194, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  46. [46]

    A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations,

    M. T. R. Laskar, S. Alqahtani, M. S. Bari, M. Rahman, M. A. M. Khan, H. Khan, I. Jahan, A. Bhuiyan, C. W. Tan, M. R. Parvezet al., “A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 1...

    work page 2024

  47. [47]

    Flexi: Benchmarking full-duplex human-llm speech interaction,

    Y . Ge, S. Chen, J. Xiao, X. Liu, T. Xiao, Y . Xiang, Z. Yu, and J. Zhu, “Flexi: Benchmarking full-duplex human-llm speech interaction,”arXiv preprint arXiv:2509.22243, 2025

    work page arXiv 2025

  48. [48]

    Speech to Speech AI Model & Provider Leaderboard,

    Artificial Analysis, “Speech to Speech AI Model & Provider Leaderboard,” https://artificialanalysis.ai/speech-to-speech? conversational-dynamics=blended-score, 2026, accessed: 2026-06- 30

    work page 2026

  49. [49]

    Speech Arena Leaderboard,

    Speech Arena, “Speech Arena Leaderboard,” https://www.speecharena. org/leaderboard, 2026, accessed: 2026-06-30

    work page 2026

  50. [50]

    Methodology — V oice Arena TTS Leaderboard,

    V oice Arena, “Methodology — V oice Arena TTS Leaderboard,” https: //voicearena.com/tts-methodology, 2026, accessed: 2026-06-30

    work page 2026

  51. [51]

    Robust speech recognition via large-scale weak supervi- sion,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    work page 2023

  52. [52]

    Silero vad: pre-trained enterprise-grade voice activity detec- tor (vad), number detector and language classifier,

    S. Team, “Silero vad: pre-trained enterprise-grade voice activity detec- tor (vad), number detector and language classifier,” https://github.com/ snakers4/silero-vad, 2024

    work page 2024

  53. [53]

    Scaling speech technology to 1,000+ languages,

    V . Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi, A. Baevski, Y . Adi, X. Zhang, W.-N. Hsu, A. Conneau, and M. Auli, “Scaling speech technology to 1,000+ languages,”arXiv, 2023

    work page 2023

  54. [54]

    Fleurs: Few-shot learning evaluation of universal representations of speech,

    A. Conneau, M. Ma, S. Khanuja, Y . Zhang, V . Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” in2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 798–805

    work page 2023

  55. [55]

    Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. [Online]. Available: http: //arxiv.org/abs/1908.10084

    work page internal anchor Pith review Pith/arXiv arXiv 2019