Pith. sign in

REVIEW 3 major objections 2 minor 1 cited by

No voice agent exceeds 0.5 on both accuracy and experience metrics at once.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 21:16 UTC pith:GXZWOMWX

load-bearing objection EVA-Bench gives a concrete simulation setup and two composite metrics for voice agents, but the bot-to-bot conversations are not checked against real human sessions. the 3 major comments →

arxiv 2605.13841 v2 pith:GXZWOMWX submitted 2026-05-13 cs.SD cs.AIcs.CLcs.LG

EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

Tara Bogavelli , Gabrielle Gauthier Melan\c{c}on , Katrina Stankiewicz , Oluwanifemi Bamgbose , Fanny Riols , Hoang H. Nguyen , Raghav Mehndiratta , Lindsay Devon Brin
show 5 more authors
Joseph Marinier Hari Subramani Anil Madamala Sridhar Krishna Nemala Srinivas Sunkara
This is my paper
classification cs.SD cs.AIcs.CLcs.LG
keywords voice agentsevaluation benchmarkconversational AIsimulation frameworkaccuracy metricexperience metricrobustness evaluationenterprise applications
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces EVA-Bench to evaluate voice agents by generating and validating multi-turn bot-to-bot audio conversations across 213 enterprise scenarios. It defines two composite metrics, EVA-A to measure task completion, faithfulness and speech fidelity, and EVA-X to measure conversation flow, conciseness and timing. When run on 12 systems from all major architectures, the results show no system clears 0.5 on both pass@1 scores, a median 0.44 gap between peak and consistent performance on EVA-A, and clear drops under accent and noise changes. The framework therefore supplies a single testbed that can compare architectures directly and expose where current designs fall short on reliability and robustness.

Core claim

EVA-Bench orchestrates bot-to-bot audio conversations over dynamic multi-turn dialogues with automatic validation that regenerates invalid simulations before scoring. It applies EVA-A, which captures task completion, faithfulness and audio-level speech fidelity, and EVA-X, which captures conversation progression, spoken conciseness and turn-taking timing, to all major agent architectures. Across the 12 tested systems, no system exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; the median pass@k minus pass^k gap on EVA-A is 0.44; and accent and noise perturbations produce mean score drops up to 0.314 that vary by architecture, system and metric.

What carries the argument

EVA-Bench's bot-to-bot simulation engine with automatic validation, together with the EVA-A accuracy metric and EVA-X experience metric.

Load-bearing premise

Automatically validated bot-to-bot audio simulations produce conversations whose failure modes and quality judgments match those of real human users interacting with deployed voice agents.

What would settle it

A side-by-side study in which the same 12 systems are evaluated by real human users on the same scenarios, then checking whether the relative scores, failure types and robustness patterns align with the EVA-Bench results.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Voice agent designs must accept a trade-off between task accuracy and conversational experience.
  • Single best-run performance overstates what a system can deliver reliably across repeated attempts.
  • Accent and noise robustness must be addressed separately for each architecture because degradation patterns differ.
  • The same set of scenarios and metrics can now be used to compare agents built on any of the three main architectures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the simulation results hold for human users, most current enterprise voice agents will still require substantial work before they can handle varied real-world conditions without frequent failures.
  • The benchmark data could serve as training signals to improve turn-taking and conciseness in future agent models.
  • Extending the perturbation suite to other real-world variables such as background speech or device distortion would test whether the observed robustness gaps are even wider.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper introduces EVA-Bench, an end-to-end framework for evaluating voice agents. It uses bot-to-bot audio simulations of dynamic multi-turn dialogues with automatic error detection and regeneration, across 213 scenarios in three enterprise domains. Two new composite metrics are defined: EVA-A (Accuracy), combining task completion, faithfulness, and audio speech fidelity; and EVA-X (Experience), combining conversation progression, spoken conciseness, and turn-taking timing. The framework is applied to 12 systems spanning three architectures, reporting pass@1, pass@k, and pass^k results plus controlled accent/noise perturbations. Key findings are that no system exceeds 0.5 on both EVA-A and EVA-X pass@1, the median pass@k–pass^k gap on EVA-A is 0.44, and perturbations produce robustness deltas up to 0.314, with variation across architectures.

Significance. If the bot-to-bot simulations are shown to produce failure modes and quality judgments that correlate with real human interactions, EVA-Bench would supply a much-needed standardized, open-source benchmark for cross-architecture voice-agent evaluation that explicitly separates accuracy from user-experience dimensions and distinguishes peak from reliable performance. The release of the full framework, evaluation suite, and benchmark data is a concrete strength that supports reproducibility.

major comments (3)
  1. [simulation engine and metric definitions] Simulation engine and metric definitions: the headline claims (no system simultaneously >0.5 on both EVA-A/EVA-X pass@1; median 0.44 gap; robustness deltas up to 0.314) rest entirely on scores derived from automatically validated bot-to-bot audio dialogues, yet the manuscript reports no human-subject study, correlation analysis, or side-by-side failure-mode comparison with real-user sessions. Without such validation, the cross-architecture rankings and robustness conclusions cannot be assumed to transfer.
  2. [EVA-A / EVA-X metric definitions] EVA-A and EVA-X definitions: the abstract states concrete numerical findings but supplies no information on how the composite metrics were constructed (component weights, normalization, aggregation rules). This information is required to interpret the absolute values and to assess whether the reported gaps are robust to reasonable reweightings.
  3. [benchmark construction] Scenario selection (213 scenarios): it is not stated whether the scenarios were chosen independently of the 12 systems under test or whether any filtering or design choices were informed by preliminary runs on those systems; dependence would undermine the generality of the performance and robustness claims.
minor comments (2)
  1. [evaluation protocol] The notation pass^k is used in the abstract and findings but its precise definition (distinct from pass@k) is not restated in the provided text; a short clarifying sentence or equation would improve readability.
  2. [robustness evaluation] The perturbation suite (accent and noise) is described at a high level; adding a table listing the exact perturbation parameters and their application order would aid reproducibility.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for the constructive feedback. We address each major comment below. We will revise the manuscript to clarify metric construction and scenario independence, and add discussion of simulation limitations. The lack of human validation is noted as a standing limitation.

read point-by-point responses

  1. Referee: Simulation engine and metric definitions: the headline claims (no system simultaneously >0.5 on both EVA-A/EVA-X pass@1; median 0.44 gap; robustness deltas up to 0.314) rest entirely on scores derived from automatically validated bot-to-bot audio dialogues, yet the manuscript reports no human-subject study, correlation analysis, or side-by-side failure-mode comparison with real-user sessions. Without such validation, the cross-architecture rankings and robustness conclusions cannot be assumed to transfer.

    Authors: We acknowledge that the paper does not include human-subject validation or correlation analysis with real-user sessions. The bot-to-bot engine with automatic error detection is intended to enable scalable evaluation of dynamic dialogues, but we agree this does not substitute for human correlation. We will add an explicit limitations subsection discussing the simulation assumptions and outlining future validation plans, but cannot provide such a study in the current work. revision: partial

  2. Referee: EVA-A / EVA-X metric definitions: the abstract states concrete numerical findings but supplies no information on how the composite metrics were constructed (component weights, normalization, aggregation rules). This information is required to interpret the absolute values and to assess whether the reported gaps are robust to reasonable reweightings.

    Authors: The full manuscript (Section 3) details the EVA-A and EVA-X construction, including the specific components (task completion, faithfulness, speech fidelity for EVA-A; progression, conciseness, turn-taking for EVA-X), their weights, normalization, and aggregation into the composite scores. We will revise the abstract to briefly list the components and direct readers to the methods for weights and rules. revision: yes

  3. Referee: Scenario selection (213 scenarios): it is not stated whether the scenarios were chosen independently of the 12 systems under test or whether any filtering or design choices were informed by preliminary runs on those systems; dependence would undermine the generality of the performance and robustness claims.

    Authors: The 213 scenarios were developed from real enterprise use cases across the three domains independently of the 12 evaluated systems; no preliminary runs on those systems informed scenario design or filtering. We will add an explicit statement of this independence in the benchmark construction section of the revised manuscript. revision: yes

standing simulated objections not resolved
  • The absence of a human-subject study, correlation analysis, or side-by-side comparison validating bot-to-bot simulations and metrics against real-user interactions.

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper defines a new benchmark framework, simulation engine, and composite metrics EVA-A and EVA-X, then reports direct empirical measurements of system performance on those definitions. No equations, fitted parameters, or self-citations are used to derive the headline numerical claims (e.g., no system exceeds 0.5 on both pass@1 scores, median gap of 0.44); those claims are measurements on the released benchmark rather than reductions of prior results. The central claims therefore remain independent of any self-referential loop.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 2 invented entities

The framework rests on the untested premise that bot-to-bot audio dialogues reproduce the distribution of real-user failure modes; the two composite metrics are newly invented constructs whose internal weighting and aggregation rules are not specified in the abstract.

axioms (1)
  • domain assumption Bot-to-bot audio conversations with automatic validation can generate realistic multi-turn test cases whose quality distribution matches human-agent interactions.
    Central to the simulation engine described in the abstract.
invented entities (2)
  • EVA-A composite metric no independent evidence
    purpose: Aggregates task completion, faithfulness, and audio-level speech fidelity into a single accuracy score.
    Newly introduced in the paper; no independent evidence supplied.
  • EVA-X composite metric no independent evidence
    purpose: Aggregates conversation progression, spoken conciseness, and turn-taking timing into a single experience score.
    Newly introduced in the paper; no independent evidence supplied.

pith-pipeline@v0.9.1-grok · 5902 in / 1574 out tokens · 32784 ms · 2026-06-30T21:16:17.113275+00:00 · methodology

0 comments
read the original abstract

Voice agents, artificial intelligence systems that conduct spoken conversations to complete tasks, are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses two core evaluation challenges: generating realistic simulated conversations, and measuring quality across the full scope of voice-specific failure modes. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates bot-to-bot audio conversations over dynamic multi-turn dialogues, with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy), capturing task completion, faithfulness, and audio-level speech fidelity; and EVA-X (Experience), capturing conversation progression, spoken conciseness, and turn-taking timing. Both metrics apply to all major agent architectures, enabling direct cross-architecture comparison. EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and pass@1, pass@k, pass^k measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k--pass^k gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean $\Delta$ up to 0.314). We release the full framework, evaluation suite, and benchmark data under an open-source license.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Just A Rather Very Intelligent Spoken Agent

    cs.AI 2026-07 conditional novelty 6.0

    JarvisBench, a two-track benchmark with a reference spoken mediator, shows mediation can raise WildClaw task scores by up to 11.78 points and improve trace-grounded user answers.

Reference graph

Works this paper leans on

138 extracted references · 138 canonical work pages · cited by 1 Pith paper · 4 internal anchors

  1. [1]

    Andres, Vadim Fedorov, Rida Sadek, Enric Spagnolo-Arrizabalaga, and Nadescha Trudel

    Miguel E. Andres, Vadim Fedorov, Rida Sadek, Enric Spagnolo-Arrizabalaga, and Nadescha Trudel. Testing the Testers: Human-Driven Quality Assessment of Voice AI Testing Platforms. 2026. URLhttps://arxiv.org/abs/ 2511.04133. 10

    work page arXiv 2026

  2. [2]

    SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words

    Junyi Ao, Yuancheng Wang, Xiaohai Tian, Dekun Chen, Jun Zhang, Lu Lu, Yuxuan Wang, Haizhou Li, and Zhizheng Wu. SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=PnjbvbblGv

    work page 2024

  3. [3]

    Talking Turns: Bench- marking Audio Foundation Models on Turn-Taking Dynamics

    Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu, Ruoming Pang, and Shinji Watanabe. Talking Turns: Bench- marking Audio Foundation Models on Turn-Taking Dynamics. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=2e4ECh0ikn

    work page 2025

  4. [4]

    Beyond task completion: Revealing corrupt success in llm agents through procedure- aware evaluation,

    Hongliu Cao, Ilias Driouich, and Eoin Thomas. Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation. 2026. URLhttps://arxiv.org/abs/2603.03116

    work page arXiv 2026

  5. [5]

    Tan, and Haizhou Li

    Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. VoiceBench: Benchmarking LLM-Based Voice Assistants.Transactions of the Association for Computational Linguistics, 14:378–398, 04 2026. ISSN 2307-387X. doi: 10.1162/TACL.a.628. URLhttps://doi.org/10.1162/TACL.a.628

    work page doi:10.1162/tacl.a.628 2026

  6. [6]

    VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models

    Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, and Irwin King. VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.),Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp...

    work page doi:10.18653/v1/2025.acl-long.818 2025

  7. [7]

    Pipecat: Open Source Framework for Voice and Multimodal Conversational AI.https://github.com/ pipecat-ai/pipecat, 2024

    Daily. Pipecat: Open Source Framework for Voice and Multimodal Conversational AI.https://github.com/ pipecat-ai/pipecat, 2024. Accessed: 2026

    work page 2024

  8. [8]

    ElevenLabs: Documentation.https://elevenlabs.io/docs/eleven-agents/overview, 2023

    ElevenLabs. ElevenLabs: Documentation.https://elevenlabs.io/docs/eleven-agents/overview, 2023. Ac- cessed: 2026

    work page 2023

  9. [9]

    Gemini Live API: Low-Latency Bidirectional Voice and Video Interactions.https://ai

    Google DeepMind. Gemini Live API: Low-Latency Bidirectional Voice and Video Interactions.https://ai. google.dev/gemini-api/docs/live, 2024. Accessed: 2026

    work page 2024

  10. [10]

    Ryan, Aditya Shrivastava, Ali Sartaz Khan, Caleb Ziems, Ella Li, Martijn Bartelds, Michael Sun, Tan Li, Woody Gan, and Diyi Yang

    Will Held, Michael J. Ryan, Aditya Shrivastava, Ali Sartaz Khan, Caleb Ziems, Ella Li, Martijn Bartelds, Michael Sun, Tan Li, Woody Gan, and Diyi Yang. CAVA: Comprehensive Assessment of Voice Assistants. https://github.com/SALT-NLP/CAVA, 2025. URLhttps://talkarena.org/cava. A benchmark for evaluating large audio models (LAMs) capabilities across six domai...

    work page 2025

  11. [11]

    Detection thresholds for gaps, overlaps, and no-gap-no-overlaps.The Journal of the Acoustical Society of America, 130(1):508–513, 2011

    Mattias Heldner. Detection thresholds for gaps, overlaps, and no-gap-no-overlaps.The Journal of the Acoustical Society of America, 130(1):508–513, 2011. doi: 10.1121/1.3598457

    work page doi:10.1121/1.3598457 2011

  12. [12]

    Pauses, gaps and overlaps in conversations.Journal of Phonetics, 38(4):555–568,

    Mattias Heldner and Jens Edlund. Pauses, gaps and overlaps in conversations.Journal of Phonetics, 38(4):555–568,

    work page

  13. [13]

    doi: https://doi.org/10.1016/j.wocn.2010.08.002

    ISSN 0095-4470. doi: https://doi.org/10.1016/j.wocn.2010.08.002. URLhttps://www.sciencedirect.com/ science/article/pii/S0095447010000628

    work page doi:10.1016/j.wocn.2010.08.002 2010

  14. [14]

    The VoiceMOS Challenge 2022

    Wen-Chin Huang, Erica Cooper, Yu Tsao, Hsin-Min Wang, Tomoki Toda, and Junichi Yamagishi. The VoiceMOS Challenge 2022. InInterspeech 2022, pp. 4536–4540, 2022. doi: 10.21437/Interspeech.2022-970

    work page doi:10.21437/interspeech.2022-970 2022

  15. [15]

    SHEET: A Multi-purpose Open-source Speech Human Evaluation Estimation Toolkit

    Wen-Chin Huang, Erica Cooper, and Tomoki Toda. SHEET: A Multi-purpose Open-source Speech Human Evaluation Estimation Toolkit. InInterspeech 2025, pp. 2355–2359, 2025. doi: 10.21437/Interspeech.2025-1977

    work page doi:10.21437/interspeech.2025-1977 2025

  16. [16]

    VoiceAgentBench: Are Voice Assistants ready for agentic tasks?arXiv preprint arXiv:2510.07978, 2026

    Dhruv Jain, Harshit Shukla, Gautam Rajeev, Ashish Kulkarni, Chandra Khatri, and Shubham Agarwal. VoiceAgentBench: Are Voice Assistants ready for agentic tasks?arXiv preprint arXiv:2510.07978, 2026. doi: 10.48550/arXiv.2510.07978

    work page doi:10.48550/arxiv.2510.07978 2026

  17. [17]

    Levinson and Francisco Torreira

    Stephen C. Levinson and Francisco Torreira. Timing in turn-taking and its implications for processing models of language.Frontiers in Psychology, 6:731, 2015. doi: 10.3389/fpsyg.2015.00731

    work page doi:10.3389/fpsyg.2015.00731 2015

  18. [18]

    Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner

    Guan-Ting Lin, Shih-Yun Shan Kuan, Jiatong Shi, Kai-Wei Chang, Siddhant Arora, Shinji Watanabe, and Hung-yi Lee. Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner.arXiv preprint arXiv:2510.07838, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  19. [19]

    Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities

    Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H Liu, and Hung-yi Lee. Full-duplex-bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities. arXiv preprint arXiv:2503.04721, 2025. 11

    work page Pith review arXiv 2025

  20. [20]

    Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency

    Guan-Ting Lin, Chen Chen, Zhehuai Chen, and Hung-yi Lee. Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency.arXiv preprint arXiv:2604.04847, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  21. [21]

    Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models

    Guan-Ting Lin, Shih-Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li, Shinji Watanabe, and Hung-yi Lee. Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models. InICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 19447–19451. IEEE,

    work page 2026

  22. [22]

    doi: 10.48550/arXiv.2507.23159

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2507.23159

  23. [23]

    EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge

    Ruskin Raj Manku, Yuzhi Tang, Xingjian Shi, Mu Li, and Alexander Smola. EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (eds.),Advances in Neural Information Processing Systems, volume 38. Curran...

    work page 2025

  24. [24]

    Beyond accuracy: A multi-dimensional framework for evaluating enterprise agentic ai systems,

    Sushant Mehta. Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems. arXiv preprint arXiv:2511.14136, 2025

    work page arXiv 2025

  25. [25]

    AI Voice Agents: 2025 Update

    Olivia Moore. AI Voice Agents: 2025 Update. https://a16z.com/ai-voice-agents-2025-update/, 2025. Andreessen Horowitz

    work page 2025

  26. [26]

    Realtime API Documentation.https://platform.openai.com/docs/guides/realtime, 2024

    OpenAI. Realtime API Documentation.https://platform.openai.com/docs/guides/realtime, 2024. Accessed: 2026

    work page 2024

  27. [27]

    Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. InProceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 ofPMLR, pp. 48371–48392, 2025

    work page 2025

  28. [28]

    FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systems

    Yizhou Peng, Yi-Wen Chao, Dianwen Ng, Yukun Ma, Chongjia Ni, Bin Ma, and Eng Siong Chng. FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systems. InProc. Interspeech 2025, pp. 176–180, 2025

    work page 2025

  29. [29]

    2508.15432 , archivePrefix =

    Bidyapati Pradhan, Surajit Dasgupta, Amit Kumar Saha, Omkar Anustoop, Sriram Puttagunta, Vipul Mittal, and Gopal Sarda. SyGra: A Unified Graph-Based Framework for Scalable Generation, Quality Tagging, and Management of Synthetic Data. 2025. URLhttps://arxiv.org/abs/2508.15432

    work page arXiv 2025

  30. [30]

    ToolLLM: Facilitating large language models to master 16000+ real-world APIs

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URLhttps://openreview.net...

    work page 2024

  31. [31]

    τ-voice: Benchmarking full-duplex voice agents on real-world domains,

    Soham Ray, Keshav Dhandhania, Victor Barres, and Karthik Narasimhan.τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains, 2026. URLhttps://arxiv.org/abs/2603.13686

    work page arXiv 2026

  32. [32]

    Judgments Concerning the Valence of Inter-Turn Silence Across Speakers of American English, Italian, and Japanese.Discourse Processes, 48(5):331–354, 2011

    Felicia Roberts, Piera Margutti, and Shoji Takano. Judgments Concerning the Valence of Inter-Turn Silence Across Speakers of American English, Italian, and Japanese.Discourse Processes, 48(5):331–354, 2011. doi: 10.1080/0163853X.2011.558002

    work page doi:10.1080/0163853x.2011.558002 2011

  33. [33]

    Turn-Taking in Conversational Systems and Human-Robot Interaction: A Review.Computer Speech & Language, 67:101178, 2021

    Gabriel Skantze. Turn-Taking in Conversational Systems and Human-Robot Interaction: A Review.Computer Speech & Language, 67:101178, 2021. URLhttps://arxiv.org/abs/2010.03674

    work page arXiv 2021

  34. [34]

    Universals and cultural variation in turn-taking in conversation.Proceedings of the National Academy of Sciences, 106(26):10587–10592, 2009

    Tanya Stivers, Nicholas J Enfield, Penelope Brown, Christina Englert, Makoto Hayashi, Trine Heinemann, Gertie Hoymann, Federico Rossano, Jan Peter De Ruiter, Kyung-Eun Yoon, et al. Universals and cultural variation in turn-taking in conversation.Proceedings of the National Academy of Sciences, 106(26):10587–10592, 2009

    work page 2009

  35. [35]

    SUPERB: Speech Processing Universal PERformance Benchmark

    Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al. SUPERB: Speech Processing Universal PERformance Benchmark. InProc. Interspeech 2021, pp. 1194–1198, 2021

    work page 2021

  36. [36]

    InThe Thirteenth International Conference on Learning Representations,

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan.τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. InThe Thirteenth International Conference on Learning Representations,

    work page

  37. [37]

    URLhttps://openreview.net/forum?id=roNSXZpUDN

    work page

  38. [38]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM- as-a-Judge with MT-Bench and Chatbot Arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, 12 M. Hardt, and S. Levine (eds.),Advances in Neural Information Processi...

    work page 2023

  39. [39]

    Available: https://arxiv.org/abs/2603.11245

    Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie, Jiarui Liu, Weihua Du, Sean Welleck, Yiming Yang, Graham Neubig, Sherry Tongshuang Wu, and Maarten Sap. Mind the Sim2Real Gap in User Simulation for Agentic Tasks. arXiv preprint arXiv:2603.11245, 2026. 13 A Definitions & Key Terms STT Speech-to-Text.A model or service that transcribes spoken audio into text....

    work page internal anchor Pith review arXiv 2026

  40. [40]

    Policy specification.Domain policies and workflow constraints are defined and reviewed prior to generation

    work page

  41. [41]

    Joint generation.SyGra generates user goals, initial databases, and expected final states jointly from a workflow graph, using GPT-5.2 as the generative backbone

    work page

  42. [42]

    Multi-intent composition.Multi-intent scenarios are constructed by combining single-intent records into coherent multi-workflow user goals, with expected final states merged accordingly

    work page

  43. [43]

    End Conversation

    Adversarial scenario design.Adversarial scenarios are hand-designed around specific policy boundary conditions, then verified against tool executor behavior to confirm that the policy violation is achievable but detectable by a correctly behaving agent. Human Review Following generation, all scenarios went through multiple rounds of manual review. Reviewe...

    work page

  44. [44]

    The agent has confirmed your request is resolved (all steps are completed) and you have said goodbye

    work page

  45. [45]

    The agent has initiated a transfer to a live agent

    work page

  46. [46]

    The agent has been unable to make progress for at least 5 consecutive turns

    work page

  47. [47]

    The agent says goodbye or indicates the conversation is over

    work page

  48. [48]

    The agent indicates that the remainder of your request cannot be fulfilled

    work page

  49. [49]

    I’m sorry I encountered an error processing your request

    If the assistant says something along the lines of "I’m sorry I encountered an error processing your request." 20 IMPORTANT: never call this tool in the same turn that you provide the agent with data, an identifier, a request to transfer to a live agent, an approval to proceed, or any kind of additional information. Before calling this tool, always say a ...

    work page

  50. [50]

    The user simulator prompt explicitly instructs the user to decline any such offers from the agent, but we check for violations regardless

    Extra modifications.The user makes requests beyond its stated goal that invoke modification tools writing to the scenario database. The user simulator prompt explicitly instructs the user to decline any such offers from the agent, but we check for violations regardless

    work page

  51. [51]

    If the user hangs up prematurely—for example, providing actionable information and ending the call in the same turn—the agent has no opportunity to execute the required tool calls

    Premature ending.In our simulations, the user is responsible for ending the call once its goal is complete. If the user hangs up prematurely—for example, providing actionable information and ending the call in the same turn—the agent has no opportunity to execute the required tool calls. We therefore verify that the user does not terminate the conversatio...

    work page

  52. [52]

    Missing information.If the user fails to provide information the agent needs to complete the task, the evaluation is corrupted since task success cannot reasonably be expected. 21

    work page

  53. [53]

    The agent then acts on the duplicate request, causing redundant writes to the scenario database that cause the final state comparison to fail

    Duplicate modifications.Occasionally, the user simulator (particularly when using non-primary models) enters a loop and repeats requests the agent has already fulfilled. The agent then acts on the duplicate request, causing redundant writes to the scenario database that cause the final state comparison to fail

    work page

  54. [54]

    accept the earlier flight if the price difference is under $200, otherwise decline

    Decision tree violations.Each user is given a structured decision tree governing how to navigate choices during the interaction (e.g.,“accept the earlier flight if the price difference is under $200, otherwise decline”). We verify that the user adheres to this logic, since deviations would cause the agent to reach a final state inconsistent with the groun...

    work page

  55. [55]

    S2S.There is no separable TTS step, so the framework log carries notts_text or llm_response records and is dropped from the merge. Consequentlyintended_assistant_turns is left empty — S2S models typically do not expose any separate text intent — and the assistant’s entries inconversation_- trace are sourced from ElevenLabsassistant_speech (transcribed) ra...

    work page

  56. [56]

    Hybrid.The framework log is populated withtts_text or llm_response (depending on the backend), and intended_assistant_turns is built as in cascade. On the input side, however, hybrid audio-native models bypass the agent’s STT — as in S2S — so the audit-log user transcripts are unreliable, and the user entries inconversation_trace are again sourced fromuse...

    work page

  57. [57]

    stay open

    Cascade.All three streams are used unmodified: audit-log user transcripts feed bothtranscribed_- user_turns and the trace, the framework log supplies the assistant’s intended text, and ElevenLabs supplies the user’s intended text and the assistant’s transcribed text. A final post-processing step (i) aligns the per-turn dictionaries so that all sources sha...

    work page

  58. [58]

    Cascade and Hybrid.Both architectures expose anintendedtext-side reference for the assistant, i.e., the LLM’s text output before TTS (intended_assistant_turns E.1). The judge task is a direct word-for-word comparison: did the audio reproduce the intended text, with particular attention to TTS-critical entities (confirmation codes, flight numbers, dollar a...

    work page

  59. [59]

    [Assistant speaks]

    S2S.S2S systems do not typically expose any text-side intent, so there is nothing to compare the audio against in the cascade sense. We instead reformulate the question as anentity articulationcheck: does the assistant clearly and correctly speak the entities it was supposed to convey? The judge receives aredacted conversation tracein which assistant entr...

    work page 2026

  60. [60]

    If the agent asks for verification details, provide your confirmation code and last name exactly as given in the required information, then wait for the agent to read back your reservation and confirm it is yours; if they read back a different name or itinerary, correct them and re-provide the details

    work page

  61. [61]

    When the agent offers earlier-flight options, evaluate each option against ALL must-have criteria: (a) date is 2026-06-18, (b) LAX departure time is before 2:00PMPT, (c) direct LAX→SFO, (d) same-day change fee is under $80

    work page 2026

  62. [62]

    If both an 11:00AM and a 1:00PM direct option meet all must-haves, choose the earliest departure (11:00AM)

    work page

  63. [63]

    If only one option meets all must-haves, accept that option

    work page

  64. [64]

    What will the change fee be in total?

    Before the agent finalizes anything, if the agent has not clearly stated the exact same-day change fee amount, ask:“What will the change fee be in total?”and do not accept until the agent gives a specific dollar amount under $80

    work page

  65. [65]

    It needs to be today, direct LAX to SFO, leaving before 2PM, and the fee has to be under $80—can you check again?

    If the agent proposes any option that departs at or after 2:00PM, has a connection, changes airports, or has a fee of $80 or more, reject it and restate the must-haves once:“It needs to be today, direct LAX to SFO, leaving before 2PM, and the fee has to be under $80—can you check again?”

    work page

  66. [66]

    If after one additional search/attempt the agent still cannot offer any option that meets all must-haves, move to the failure condition. Resolution Condition.The agent has confirmed the rebooking is completed (not just planned) to a direct LAX→SFO flight departing on 2026-06-18 before 2:00PMPT, has stated the same-day change fee is under $80, AND has prov...

    work page 2026

  67. [68]

    Never invent new goals, requests, or problems beyond what is defined here

    work page

  68. [70]

    If the agent suggests flying from or to a different airport than originally booked, decline and insist on LAX to SFO only

    work page

  69. [71]

    V O R J U

    If the agent suggests standby instead of a confirmed earlier flight, decline standby and ask for a confirmed seat on an earlier direct flight before 2:00PM. Expected Flow, Database & Ground Truth Expected Flow.Passenger wants to move to an earlier departure on the same date. Agent applies same-day change fee ($75, waived for Gold+) and searches for earlie...

    work page 2026

  70. [72]

    (fare $228 in main cabin)

    One-stop option– depart at nine twenty a.m., connect in San Jose and arrive at twelve ten p.m. (fare $228 in main cabin)

    work page

  71. [73]

    LAX”, destination: “SFO

    Direct flight– depart at one o’clock p.m., arrive at two twenty-five p.m. (fare $289, same as your current ticket). 3.Direct flight– depart at two forty p.m., arrive at four oh-five p.m. (fare $259, a little cheaper). Because this is a voluntary same-day change, achange fee of seventy-five dollarsapplies. If you choose a lower-priced flight, the fare diff...

    work page 2026

  72. [74]

    Provide your employee ID and the last four digits of your phone number

    Start by completing identity verification only when asked. Provide your employee ID and the last four digits of your phone number. Do not volunteer other details before the agent asks

    work page

  73. [75]

    Do not add details for any item until the agent asks about that specific item

    After verification, give a brief overview of all four items: email seems down for everyone, your AD account is locked, you need Confluence access, and you want a 30-day Figma trial. Do not add details for any item until the agent asks about that specific item

    work page

  74. [76]

    If asked which service, say email

    First intent — email outage.Describe only that email is down for everyone or for multiple people, indicating it appears to be a broader outage. If asked which service, say email. Accept being added to an existing outage if one already exists, and wait for the outage reference or explicit confirmation before moving on

    work page

  75. [77]

    What happens next?

    Second intent — AD lockout.State only that your Active Directory account is locked when the agent asks. If the agent says the account cannot be unlocked because of a security hold, ask exactly one follow-up question:“What happens next?”If they explain that a ticket has been opened and provide the ticket number and expected response time or SLA, accept tha...

    work page

  76. [78]

    If asked for access level, choose read_only

    Third intent — Confluence access.Provide the application name only when asked: Confluence. If asked for access level, choose read_only. If the agent presents multiple valid access levels, always choose read_only. Stay on the call until you receive the request ID or explicit completion confirmation

    work page

  77. [79]

    If asked whether you want permanent or temporary, choose temporary

    Fourth intent — Figma trial.Provide the product name only when asked: Figma. If asked whether you want permanent or temporary, choose temporary. If asked for duration, choose 30 days. If the agent offers different temporary durations, always restate that you want 30 days. Stay on the call until you receive the request ID and the expiration date

    work page

  78. [80]

    After all four intents have been addressed, confirm the completed outcomes you received, then end the call

    work page

  79. [81]

    Do not invent missing details

    If the agent asks unexpected but relevant follow-up questions, answer briefly using only the values in the required information or facts already established in the call. Do not invent missing details. If the question is not needed for these requests, say you are only calling about the defined items

    work page

  80. [82]

    If it does not match, correct only the incorrect field and nothing else

    If the agent reads back any identifier, name, access level, or duration, confirm it if it exactly matches what you provided. If it does not match, correct only the incorrect field and nothing else. Resolution Condition.You have clear confirmation that you were added to the existing email outage or have been given the outage ticket number, you have receive...

    work page 2026

Showing first 80 references.