pith. sign in

arxiv: 2604.15710 · v1 · submitted 2026-04-17 · 💻 cs.SD

VoxMind: An End-to-End Agentic Spoken Dialogue System

Tianle Liang , Yifu Chen , Shengpeng Ji , Yijun Chen , Zhiyang Jia , Jingyu Lu , Fan Zhuo , Xueyi Pu
show 2 more authors
Yangzhuo Li Zhou Zhao
This is my paper

Pith reviewed 2026-05-10 08:26 UTC · model grok-4.3

classification 💻 cs.SD
keywords modelsspokenvoxmindagentagenticdialogueend-to-endtasks
0
0 comments X

The pith

VoxMind equips spoken dialogue models with agentic tool-use via a curated dataset, Think-before-Speak reasoning, and multi-agent tool management, raising task completion from 34.88% to 74.57% while preserving conversational quality.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Spoken dialogue systems let people talk naturally to AI, but they often cannot handle requests that need external information or actions. VoxMind tries to fix this by giving the model agent-like abilities to use tools. The team built a 470-hour dataset called AgentChat focused on agent behaviors. They added a Think-before-Speak step so the model reasons internally before speaking. To keep responses fast even with many tools, they created a multi-agent system where helper agents handle retrieval tasks in parallel. Tests showed the model completed tasks much more often, jumping from about 35 percent success with previous systems to 75 percent. It also did better than Gemini-2.5-Pro on spoken agent tasks without hurting normal conversation quality. The code and data are released publicly.

Core claim

Experimental results confirm that VoxMind achieves significant improvements in agent performance: compared with strong baselines, the task completion rate increases from 34.88% to 74.57%, outperforming Gemini-2.5-Pro on spoken agent tasks while preserving general conversational quality.

Load-bearing premise

The assumption that the reported performance gains are primarily caused by the Think-before-Speak mechanism and Multi-Agent Dynamic Tool Management rather than the specific dataset curation, unstated implementation choices, or evaluation protocol details.

Figures

Figures reproduced from arXiv: 2604.15710 by Fan Zhuo, Jingyu Lu, Shengpeng Ji, Tianle Liang, Xueyi Pu, Yangzhuo Li, Yifu Chen, Yijun Chen, Zhiyang Jia, Zhou Zhao.

Figure 1
Figure 1. Figure 1: VoxMind can dynamically perceive the inter [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overall architecture of the VoxMind. Given spoken user input, the speech-centric agent first generates an [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Dialogues demonstrating the agent’s six core capabilities. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of inference efficiency and task accuracy with and without the auxiliary LLM across varying [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Word clouds of AgentChat data: (Left) tool-interaction data; (Right) general conversational data. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Tool Interaction Data Training Example [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt for constructing chain-of-thought reasoning data for tool interaction. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt for constructing chain-of-thought reasoning data for general dialogue interaction. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt specification for evaluating the quality of Chain-of-Thought in tool-based spoken interactions. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt specification for evaluating the quality of Chain-of-Thought in general dialogue settings. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt specification for evaluating tool necessity during data cleaning. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt specification for compressing original Chain-of-Thought annotations in tool-based data cleaning. [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Prompt specification for strict, extraction-based evaluation of tool-call correctness. [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗
read the original abstract

Recent end-to-end spoken dialogue models enable natural interaction. However, as user demands become increasingly complex, models that rely solely on conversational abilities often struggle to cope. Incorporating agentic capabilities is therefore essential: by enabling tool use, these models can extend their knowledge boundaries and better solve real-world tasks. Yet, existing research has largely concentrated on core perception and generation, with comparatively limited exploration of such tool-augmented extensions. To bridge this gap, we present VoxMind, an integrated framework designed to equip end-to-end spoken dialogue models with comprehensive agentic abilities. Leveraging our curated 470-hour AgentChat dataset, we incorporate a "Think-before-Speak" mechanism, enabling the model to internalize structured reasoning as a critical prerequisite for planning and response generation. Furthermore, to mitigate latency bottlenecks caused by large-scale tool integration, we propose a Multi-Agent Dynamic Tool Management architecture. By asynchronously delegating retrieval tasks to an auxiliary agent aligned with the main model's reasoning trajectory, this system effectively decouples inference latency from toolset size. Experimental results confirm that VoxMind achieves significant improvements in agent performance: compared with strong baselines, the task completion rate increases from 34.88% to 74.57%, outperforming Gemini-2.5-Pro on spoken agent tasks while preserving general conversational quality. The source code and associated data are publicly available at https://github.com/MM-Speech/VoxMind.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Circularity Check

0 steps flagged

No circularity: empirical claims rest on reported experiments, not self-referential definitions or fits

full rationale

The paper introduces a new dataset and two architectural mechanisms, then reports task-completion improvements from end-to-end experiments. No equations, fitted parameters renamed as predictions, or self-citation chains appear in the provided text that would make any result equivalent to its inputs by construction. The performance numbers (34.88 % to 74.57 %) are presented as measured outcomes against baselines, which remain externally falsifiable and do not reduce to tautology.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available, so no explicit free parameters, axioms, or invented entities with independent evidence can be identified. The framework relies on a curated dataset and two proposed mechanisms whose details are not provided.

pith-pipeline@v0.9.0 · 5580 in / 1151 out tokens · 42515 ms · 2026-05-10T08:26:05.318408+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Survey of Audio Reasoning in Multimodal Foundation Models

    eess.AS 2026-05 unverdicted novelty 2.0

    A survey that provides a unified formulation of audio reasoning and reviews advances across Audio-to-Text, Audio-to-Speech, Audio-Visual, and Agentic paradigms while discussing challenges and future directions.

Reference graph

Works this paper leans on

44 extracted references · 44 canonical work pages · cited by 1 Pith paper

  1. [1]

    Stream rag: Instant and accurate spoken di- alogue systems with streaming tool usage.ArXiv, abs/2510.02044. Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang, Xi- quan Li, Ruiyang Xu, Zhikang Niu, Yanqiao Zhu, Yifan Yang, Zhanxun Liu, Kai Yu, Yuxuan Hu, Jinyu Li, Yan Lu, Shujie Liu, and Xie Chen. 2024a. Slam- omni: Timbre-controllable voice interaction system...

    work page arXiv 2025

  2. [2]

    No correct- ness judgment is performed at this stage

    Tool Extraction: Extract all tool calls from both the target and model outputs (including only tool names and parameter name-value pairs), ignoring textual content, formatting, spaces, quotes, and line breaks. No correct- ness judgment is performed at this stage

    work page

  3. [3]

    Tool occur- rence counts must match exactly; otherwise, evaluation stops immediately and both tool selection and parameter filling are marked in- correct

    Tool Selection Evaluation: Compare ex- tracted tool names (case-sensitive, ignoring order and leading/trailing spaces). Tool occur- rence counts must match exactly; otherwise, evaluation stops immediately and both tool selection and parameter filling are marked in- correct

    work page

  4. [4]

    Parameter names ignore case and spaces, while parame- ter values must match exactly

    Parameter Filling Evaluation: Performed only if tool selection is correct. Parameter names ignore case and spaces, while parame- ter values must match exactly. Numeric equiv- alence and quoting differences are allowed, and argument order does not affect the evalua- tion

    work page

  5. [5]

    func-select-correct

    Output Format: Evaluation results are strictly returned in JSON format, containing only two boolean fields: "func-select-correct": true|false, "param-fill-correct": true|false. This procedure ensures rigorous, reproducible evaluation and avoids direct string comparison, pro- viding a precise measure of a speech agent’s tool- call capabilities. G Training ...

    work page

  6. [6]

    Start from the User Query; do not back-solve from the answer

    work page

  7. [7]

    Explain why the selected tool is needed and appropriate

    work page

  8. [8]

    For every parameter in the tool call, explain its source and any transformation

    work page

  9. [9]

    Do not introduce assumptions not stated in the query

    work page

  10. [10]

    Produce 5–12 steps in natural language; keep within≤THINK_MAX_WORDS words

    work page

  11. [11]

    Do NOT output the tool call; output only the reasoning

    work page

  12. [12]

    think":

    searchTools() indicates that the current tool list cannot meet the task and that additional tools are needed. Output format: Output strictly one-line JSON: {"think": "..."}. No extra text. User Query: {(user_q or ”).strip()} Gold Tool Call: {(assistant_a or ”).strip()} Figure 7: Prompt for constructing chain-of-thought reasoning data for tool interaction....

    work page

  13. [13]

    Start reasoning strictly from the User Query; do not back-solve from the final answer

    work page

  14. [14]

    Ensure that each step follows causally from the previous one

    work page

  15. [15]

    Do not introduce assumptions or external knowledge not implied by the query

    work page

  16. [16]

    Focus on semantic reasoning rather than stylistic or rhetorical choices

    work page

  17. [17]

    Produce 5–12 reasoning steps in natural language; keep within≤THINK_MAX_WORDS words

    work page

  18. [18]

    The Chain-of-Thought is used for training purposes only and will not be shown to end users

    work page

  19. [19]

    think":

    Do NOT output the final answer; output only the reasoning process. Output format: Output strictly one-line JSON: {"think": "..."}. No extra text. User Query: {(user_q or ”).strip()} Gold Response: {(assistant_a or ”).strip()} Figure 8: Prompt for constructing chain-of-thought reasoning data for general dialogue interaction. Chain-of-thought quality evalua...

    work page

  20. [20]

    Candidate Chain-of-Thought Scoring criteria (very strict):

    work page

  21. [21]

    Logical soundness (0–3): Is the reasoning stepwise, coherent, and causally connected? Are any key reasoning steps missing?

    work page

  22. [22]

    Consistency with the tool call (0–3): Does the Chain-of-Thought explain why the tool is selected and the source of each tool parameter? Is it fully consistent with the Gold Tool Call?

    work page

  23. [23]

    No hallucination (0–2): Does the reasoning avoid inventing facts, assumptions, or parameters not present in the User Query or Gold Tool Call?

    work page

  24. [24]

    score": <0–10 integer or float>,

    Clarity (0–2): Is the reasoning clear, well-structured, and easy to follow as a step-by-step explanation? Final score: The final score is the sum of all criteria, ranging from 0 to 10. Output format requirements: Output JSON only; no explanations. Fields: {"score": <0–10 integer or float>, "reason": "<brief 1–2 sentence justification>"}. Inputs: User Quer...

    work page

  25. [25]

    Generated Chain-of-Thought Scoring criteria:

    work page

  26. [26]

    Correctness (0–4): Does the Chain-of-Thought logically derive the final answer? Is each reasoning step factually correct? Is the reasoning free from hallucinations or fabricated assumptions? Scoring guidelines: 4 = fully correct 3 = mostly correct with minor issues 2 = contains some errors but the final answer is still reachable 1 = reasoning is incorrect...

    work page

  27. [27]

    Relevance (0–2): Does the Chain-of-Thought remain tightly focused on the user question? Does it avoid unrelated tangents? Scoring guidelines: 2 = fully relevant 1 = partially relevant 0 = mostly irrelevant

    work page

  28. [28]

    Step quality and clarity (0–2): Are the reasoning steps clear, structured, and easy to follow? Are there any unjustified jumps in logic? Scoring guidelines: 2 = very clear 1 = acceptable clarity 0 = unclear or disorganized

    work page

  29. [29]

    Completeness (0–1): Does the Chain-of-Thought cover all necessary steps to justify the final answer? Scoring guidelines: 1 = complete 0 = missing key steps

    work page

  30. [30]

    correctness

    Brevity and conciseness (0–1): Is the Chain-of-Thought concise and free of unnecessary verbosity? Scoring guidelines: 1 = concise 0 = overly long or verbose Final score: The total score is the sum of all criteria, ranging from 0 to 10. Output format requirements (strict): Output JSON only; no additional text. Required fields: { "correctness": X, "relevanc...

    work page

  31. [31]

    Real-time information (e.g., weather, stock prices, news, events, schedules)

    work page

  32. [32]

    External knowledge not contained in general training data (e.g., private databases, personal files, proprietary datasets)

    work page

  33. [33]

    Precise numerical computation beyond mental math (e.g., long arithmetic, complex mathematical evaluation)

    work page

  34. [34]

    Retrieval of specific, unmemorized facts (e.g., obscure identifiers, URLs, tables, large codebases)

    work page

  35. [35]

    tool_necessity

    Interaction with an external environment (e.g., search engines, APIs, calculators, file operations) A tool isnot necessarywhen: - The question asks for explanations, definitions, or conceptual reasoning - The answer can be inferred using general world knowledge - The question is creative in nature (e.g., writing, storytelling, opinions, reasoning, code ex...

    work page

  36. [36]

    Preserve the logical flow of the original reasoning while condensing it into at most {num} English words

    work page

  37. [37]

    Start from the user’s intent as reflected in the original reasoning

    work page

  38. [38]

    Explicitly mention the selected tool and explain why it is appropriate

    work page

  39. [39]

    For each parameter in the tool call, briefly indicate its source from the user request or the tool-call text

    work page

  40. [40]

    Do not introduce assumptions not supported by the original reasoning

    work page

  41. [41]

    Output only the compressed reasoning

    work page

  42. [42]

    think":

    Output strictly one-line JSON: {"think": "..."} with no extra text. Inputs: Original Chain-of-Thought: {orig_think} Gold Tool Call (raw, including tags): {raw_tool_call} Figure 12: Prompt specification for compressing original Chain-of-Thought annotations in tool-based data cleaning. Gemini-2.5-flash evaluates core capabilities of end-to-end speech agents...

    work page

  43. [43]

    - Tool name = full string before ‘(’

    Tool Selection (ONLY based on extracted tool names) - Compare tool names AFTER extraction, not raw text. - Tool name = full string before ‘(’. - Tool names are case-sensitive; ignore leading/trailing spaces. - Tool occurrence counts must match exactly (order does NOT matter). - If ANY mismatch exists: * func_select_correct = false * param_fill_correct = f...

    work page

  44. [44]

    Taylor Swift

    Parameter Filling (ONLY if Tool Selection is correct) - Compare parameters ONLY within matched tools. - Parameter names ignore case and spaces. - Parameter values must match exactly (case-sensitive). - Ignore ALL quoting differences: q=’Taylor Swift’≡q="Taylor Swift"≡q=Taylor Swift - Numeric equivalence: 42≡42.0 - Argument order does NOT matter. STRICT OU...

    work page