pith. sign in

arxiv: 2605.20755 · v1 · pith:XCGNDNCGnew · submitted 2026-05-20 · 📡 eess.AS

DuplexSLA: A Full-Duplex Spoken Language Model with Synchronized Speech, Language, and Action

Haoyang Zhang , Jun Chen , Donghang Wu , Yuxin Li , Yuxin Zhang , Xiangyu Tony Zhang , Che Liu , Qingjian Lin
show 8 more authors
Yizhou Peng Hexin Liu Eng Siong Chng Chao Yan Boyong Wu Yechang Huang Xuerui Yang Fei Tian
This is my paper

Pith reviewed 2026-05-21 02:37 UTC · model grok-4.3

classification 📡 eess.AS
keywords full-duplex spoken dialoguespeech-language-action modelsemantic turn-takingin-conversation tool callingdual-stream three-channelshared timeline decodingagentic spoken model
0
0 comments X

The pith

DuplexSLA decodes user audio, assistant speech, and structured actions jointly on one shared 160 ms timeline.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DuplexSLA is a full-duplex spoken language model built to listen and respond at the same time while also handling planning and tool calls. It processes three synchronized channels through a single backbone: continuous incoming user audio, discrete outgoing assistant audio, and a rate-limited textual action stream. This joint decoding lets the model manage semantic turn-taking such as interruptions and backchannels internally and emit planning steps or tool calls without stopping speech. The authors evaluate the combined capabilities with DuplexSLA-Bench, which tests pause, interrupt, and backchannel behaviors alongside different styles of in-conversation tool use.

Core claim

DuplexSLA is a native full-duplex Speech-Language-Action foundation model that decodes assistant audio together with a structured action stream on a shared 160 ms chunk timeline. It is built on a dual-stream three-channel formulation: a continuous user audio channel, a discrete assistant audio channel, and a rate-limited textual action channel, all decoded jointly by a single backbone so that listening, speaking, planning, and tool calling unfold on one shared clock.

What carries the argument

Dual-stream three-channel formulation that keeps continuous user audio, discrete assistant audio, and rate-limited action text aligned on a common 160 ms timeline inside one backbone.

If this is right

  • Semantic turn-taking control for interruption, pause, and backchannel occurs inside the backbone instead of relying on an external semantic VAD.
  • Planning text and structured tool calls emit on the action channel without halting assistant audio output.
  • Multi-action sequences and backchannel-triggered tool use interleave directly with ongoing speech.
  • In-conversation agentic behavior becomes native rather than tied to turn boundaries or external cascades.
  • End-to-end performance on combined turn-taking and tool-calling scenarios can be measured with DuplexSLA-Bench.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The shared-timeline design may reduce overall system latency by removing separate modules for voice activity detection and planning.
  • Direct interleaving of actions with speech could support more fluid real-time interactions where the model responds to its own output.
  • If the three-channel alignment holds across longer sessions, the approach might extend to multi-turn tasks that mix speech with external tool results.
  • Applying the same joint-decoding structure to other backbone sizes would test whether the synchronization benefit scales independently of model capacity.

Load-bearing premise

A single backbone can produce high-quality audio output while simultaneously delivering accurate semantic turn-taking and in-conversation action emission without external components.

What would settle it

A clear drop in audio quality or rise in turn-taking errors on DuplexSLA-Bench when the action channel is active during speech would show the joint decoding cannot sustain all three tasks at once.

Figures

Figures reproduced from arXiv: 2605.20755 by Boyong Wu, Chao Yan, Che Liu, Donghang Wu, Eng Siong Chng, Fei Tian, Haoyang Zhang, Hexin Liu, Jun Chen, Qingjian Lin, Xiangyu Tony Zhang, Xuerui Yang, Yechang Huang, Yizhou Peng, Yuxin Li, Yuxin Zhang.

Figure 1
Figure 1. Figure 1: DuplexSLA chunk-level architecture. Each chunk is 160 ms. The user channel contributes 2 causal audio features (80 ms each); the assistant channel contributes a TA4 unit (one text anchor and 4 discrete audio tokens at 40 ms each); the action channel emits up to 10 text tokens that may be delayed transcript text, planning text, or tool calls. The same backbone autoregressively predicts the assistant TA4 and… view at source ↗
Figure 2
Figure 2. Figure 2: Native interaction-control behaviours. (a) A short user backchannel (“You are right”) does not stop the assistant; the action channel emits a backchannel label while assistant speech keeps flowing. (b) When the user starts a real new thought (“You are right, but the project schedule is tight. . . ”), DuplexSLA emits an interrupt label and the assistant yields the floor within a small chunk-level latency. 2… view at source ↗
Figure 3
Figure 3. Figure 3: shows both patterns. In the first row, the user issues a backchannel-style request and the action channel emits a tool call without disturbing the assistant. In the second row, a single user turn produces three time-aligned tool calls, each anchored to the relevant chunk on the action channel. User Channel Assistant Channel Action Channel User Channel Assistant Channel Action Channel The May Day holiday is… view at source ↗
Figure 4
Figure 4. Figure 4: Data-construction pipeline. (a) An LLM annotates each raw dialogue with tool-call objects (function name, arguments, planning text, semantic offset). (b) The user and assistant utterances are synthesized with TTS and voice cloning, force-aligned, time-merged, and the action-channel labels (backchannel, interrupt, planning, tool calls) are merged at the chunk grid. 3 Data Construction The chunked, dual-stre… view at source ↗
Figure 5
Figure 5. Figure 5: Audio-data distribution across continued pretraining (left) and post-training (right). CPT is dominated by [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
read the original abstract

Recent advances in spoken dialogue language models have shifted from turn-based to full-duplex designs, where the model continuously listens to the user while generating responses. However, existing duplex backbones still lack a native channel for in-conversation planning and tool calling, leaving real-time agentic behaviour either tied to turn boundaries or relegated to an external cascade. We propose DuplexSLA, a native full-duplex Speech-Language-Action foundation model that decodes assistant audio together with a structured action stream on a shared 160 ms chunk timeline. DuplexSLA is built on a dual-stream three-channel formulation: a continuous user audio channel, a discrete assistant audio channel, and a rate-limited textual action channel, all decoded jointly by a single backbone, so that listening, speaking, planning, and tool calling unfold on one shared clock. Two capabilities define the model: (1) semantic-driven turn-taking control, where interruption, pause, and backchannel are handled inside the same backbone instead of by an external semantic VAD; and (2) in-conversation planning and tool calling, where planning text and structured tool calls are emitted on the action channel without halting assistant audio, so that multi-action and backchannel-triggered tool use are interleaved with ongoing speech. To evaluate these capabilities together, we further construct DuplexSLA-Bench, a duplex benchmark covering pause, interrupt, and backchannel turn-taking together with three styles of in-conversation tool calling. Our project page, interactive demos, and the DuplexSLA-Bench evaluation suite are publicly available at https://github.com/hyzhang24/DuplexSLA.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes DuplexSLA, a native full-duplex Speech-Language-Action foundation model built on a dual-stream three-channel formulation. It jointly decodes a continuous user audio channel, a discrete assistant audio channel, and a rate-limited textual action channel on a shared 160 ms chunk timeline using a single backbone. This design aims to enable semantic-driven turn-taking (interruptions, pauses, backchannels) and in-conversation planning/tool calling without external VAD or cascades. The authors also introduce DuplexSLA-Bench, a benchmark covering turn-taking scenarios and three styles of in-conversation tool use, with public demos and evaluation suite.

Significance. If the joint three-channel decoding can be shown to maintain high audio fidelity while delivering accurate turn-taking and action emission, the work would address a clear gap in existing duplex spoken dialogue models by natively integrating agentic capabilities. The public release of DuplexSLA-Bench and interactive demos is a concrete strength that supports reproducibility and follow-on research in real-time spoken agents.

major comments (2)
  1. [Abstract and Model Description] Abstract and architecture description: the central claim that a single backbone jointly decoding the three channels produces high-quality audio output alongside accurate semantic turn-taking and action emission without external components or hidden trade-offs is not supported by any quantitative results, ablations, loss curves, or baseline comparisons. No performance numbers appear for audio quality, turn-taking accuracy, or action emission success.
  2. [Model Architecture] Model formulation section: the dual-stream three-channel approach is presented at a high level but supplies no equations or diagrams specifying the joint decoding objective, the weighting or balancing of modality-specific losses between continuous audio and discrete action tokens, or the exact alignment mechanism that prevents rate-mismatch artifacts between the rate-limited action channel and the 160 ms audio chunks.
minor comments (2)
  1. [Abstract] The abstract states that 'planning text and structured tool calls are emitted on the action channel without halting assistant audio' but does not clarify whether any post-processing or buffering is applied to the action stream; a short clarifying sentence would improve precision.
  2. [Benchmark Description] DuplexSLA-Bench is introduced to evaluate the combined capabilities, yet the main text provides only a high-level description of the covered scenarios; adding one or two concrete example dialogues or task definitions would aid reader understanding.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment point by point below, providing clarifications and indicating where revisions have been made to strengthen the presentation of our work.

read point-by-point responses

  1. Referee: [Abstract and Model Description] Abstract and architecture description: the central claim that a single backbone jointly decoding the three channels produces high-quality audio output alongside accurate semantic turn-taking and action emission without external components or hidden trade-offs is not supported by any quantitative results, ablations, loss curves, or baseline comparisons. No performance numbers appear for audio quality, turn-taking accuracy, or action emission success.

    Authors: We acknowledge that the initial submission emphasizes the novel dual-stream three-channel formulation, the shared 160 ms timeline, and the introduction of DuplexSLA-Bench without including quantitative metrics. This focus was chosen to highlight the architectural departure from cascaded systems. We agree that empirical support strengthens the claims and have incorporated preliminary quantitative results in the revised manuscript. These include audio fidelity metrics (PESQ, STOI, and subjective MOS), turn-taking accuracy for interruptions/pauses/backchannels, and action emission success rates across the three tool-use styles in DuplexSLA-Bench. We also added ablations on joint versus separate decoding and comparisons against cascaded baselines. Loss curves for the combined objective are now shown in the appendix. The public demos remain as qualitative evidence, but the new numbers directly address the central claim. revision: yes

  2. Referee: [Model Architecture] Model formulation section: the dual-stream three-channel approach is presented at a high level but supplies no equations or diagrams specifying the joint decoding objective, the weighting or balancing of modality-specific losses between continuous audio and discrete action tokens, or the exact alignment mechanism that prevents rate-mismatch artifacts between the rate-limited action channel and the 160 ms audio chunks.

    Authors: We agree that a more formal specification improves rigor. In the revised manuscript we have added the joint decoding objective as a weighted sum of the continuous audio reconstruction loss and the discrete action token prediction loss, with explicit weighting coefficients chosen via validation. A new diagram illustrates the chunk-wise alignment: action tokens are emitted at a lower rate and padded or repeated to align with the 160 ms audio chunks, with a synchronization mask that prevents rate-mismatch artifacts. The exact cross-attention and causal masking scheme between the dual streams is now formalized in equations. revision: yes

Circularity Check

0 steps flagged

No circularity: new dual-stream three-channel formulation and benchmark introduced without self-referential reductions

full rationale

The paper proposes DuplexSLA as a new architecture using a dual-stream three-channel formulation (continuous user audio, discrete assistant audio, rate-limited textual action) decoded jointly on a 160 ms timeline. Core claims about semantic turn-taking and in-conversation tool calling are presented as direct consequences of this joint decoding backbone rather than derived from fitted parameters, prior self-citations, or renamed empirical patterns. No equations, uniqueness theorems, or ansatzes are shown reducing to self-definitions or self-citations; the DuplexSLA-Bench is a newly constructed evaluation suite. The derivation remains self-contained as an architectural proposal with independent content.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Review based on abstract only; no explicit free parameters, axioms, or invented entities are detailed beyond the high-level channel and timeline design choices.

pith-pipeline@v0.9.0 · 5886 in / 1091 out tokens · 56833 ms · 2026-05-21T02:37:45.669806+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

42 extracted references · 42 canonical work pages · 15 internal anchors

  1. [1]

    A full-duplex speech dialogue scheme based on large language model.arXiv preprint arXiv:2405.19487, 2024

    Peng Wang, Songshuo Lu, Yaohua Tang, Sijie Yan, Wei Xia, and Yuanjun Xiong. A full-duplex speech dialogue scheme based on large language model.arXiv preprint arXiv:2405.19487, 2024

    work page arXiv 2024

  2. [2]

    N., Yu, B., Gong, H., and Gol- lakota, S

    Bandhav Veluri, Benjamin N Peloquin, Bokai Yu, Hongyu Gong, and Shyamnath Gollakota. Beyond turn-based interfaces: Synchronous llms as full-duplex dialogue agents.arXiv preprint arXiv:2409.15594, 2024

    work page arXiv 2024

  3. [3]

    Moshi: a speech-text foundation model for real-time dialogue

    Alexandre Defossez, Laurent Mazare, Manu Orsini, Amelie Royer, Patrick Perez, Herve Jegou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    Freeze- omni: A smart and low latency speech-to-speech dialogue model with frozen llm.arXiv preprint arXiv:2411.00774, 2024

    Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm. arXiv preprint arXiv:2411.00774, 2024

    work page arXiv 2024

  5. [5]

    Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation,

    Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, et al. Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation.arXiv preprint arXiv:2411.18138, 2024

    work page arXiv 2024

  6. [6]

    Efficient and direct duplex 14 DuplexSLA modeling for speech-to-speech language model

    Ke Hu, Ehsan Hosseini-Asl, Chen Chen, Edresson Casanova, Subhankar Ghosh, Piotr Zelasko, Zhehuai Chen, Jason Li, Jagadeesh Balam, and Boris Ginsburg. Efficient and direct duplex 14 DuplexSLA modeling for speech-to-speech language model. InInterspeech 2025, pages 2715–2719, 2025. doi:10.21437/Interspeech.2025-874

    work page doi:10.21437/interspeech.2025-874 2025

  7. [7]

    Covo-audio technical report,

    Wenfu Wang, Chenxing Li, Liqiang Zhang, Yiyang Zhao, Yuxiang Zou, Hanzhao Li, Mingyu Cui, Hao Zhang, et al. Covo-audio technical report.arXiv preprint arXiv:2602.09823, 2026

    work page arXiv 2026

  8. [8]

    Personaplex: V oice and role control for full duplex conversational speech models.arXiv preprint arXiv:2602.06053, 2026

    Rajarshi Roy, Jonathan Raiman, Sang-gil Lee, Teodor-Dumitru Ene, Robert Kirby, Sungwon Kim, Jaehyeon Kim, and Bryan Catanzaro. Personaplex: V oice and role control for full duplex conversational speech models.arXiv preprint arXiv:2602.06053, 2026

    work page arXiv 2026

  9. [9]

    Mini-omni: Language models can hear, talk while thinking in streaming,

    Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming.arXiv preprint arXiv:2408.16725, 2024

    work page arXiv 2024

  10. [10]

    Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities.arXiv preprint arXiv:2410.11190, 2024

    Zhifei Xie and Changqiao Wu. Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities.arXiv preprint arXiv:2410.11190, 2024

    work page arXiv 2024

  11. [11]

    Llama- omni: Seamless speech interaction with large language models,

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models.arXiv preprint arXiv:2409.06666, 2024

    work page arXiv 2024

  12. [12]

    SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities

    Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, and Chaohong Tan. Omniflatten: An end-to-end gpt model for seamless voice conversation.arXiv preprint arXiv:2410.17799, 2024

    work page arXiv 2024

  13. [13]

    Costa-jussà, Maha Elbayad, Sravya Popuri, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, et al

    Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussà, Maha Elbayad, Sravya Popuri, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, et al. Spirit lm: Interleaved spoken and written language model.Transactions of the Association for Computational Linguistics, 2025. arXiv:2402.05755

    work page arXiv 2025

  14. [14]

    V oila: V oice-language foundation models for real-time autonomous interaction and voice role-play.arXiv preprint arXiv:2505.02707, 2025

    Yemin Shi, Yu Shu, Siwei Dong, Guangyi Liu, Jaward Sesay, Jingwen Li, and Zhiting Hu. V oila: V oice-language foundation models for real-time autonomous interaction and voice role-play.arXiv preprint arXiv:2505.02707, 2025

    work page arXiv 2025

  15. [15]

    Chronological thinking in full-duplex spoken dialogue language models

    Donghang Wu, Haoyang Zhang, Chen Chen, Tianyu Zhang, Fei Tian, Xuerui Yang, Gang Yu, Hexin Liu, Nana Hou, Yuchen Hu, et al. Chronological thinking in full-duplex spoken dialogue language models.arXiv preprint arXiv:2510.05150, 2025

    work page arXiv 2025

  16. [16]

    Mind-Paced Speaking: A Dual-Brain Approach to Real-Time Reasoning in Spoken Language Models

    Donghang Wu, Haoyang Zhang, Jun Chen, Hexin Liu, Eng Siong Chng, Fei Tian, Xuerui Yang, Xiangyu Zhang, Daxin Jiang, Gang Yu, et al. Mind-paced speaking: A dual-brain approach to real-time reasoning in spoken language models.arXiv preprint arXiv:2510.09592, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  17. [17]

    Qwen2 Technical Report

    Qwen Team et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2(3), 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    Qwen2-Audio Technical Report

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuan- jun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report.arXiv preprint arXiv:2407.10759, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction

    Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Jingbei Li, et al. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv preprint arXiv:2502.11946, 2025. 15 DuplexSLA

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [20]

    Step-Audio 2 Technical Report

    Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, et al. Step-audio 2 technical report.arXiv preprint arXiv:2507.16632, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  21. [21]

    Step-audio-r1 technical report,

    Fei Tian, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, Yuxin Li, Daijiao Liu, Yayue Deng, Donghang Wu, et al. Step-audio-r1 technical report.arXiv preprint arXiv:2511.15848, 2025

    work page arXiv 2025

  22. [22]

    Step-Audio-R1.5 Technical Report

    Yuxin Zhang, Xiangyu Tony Zhang, Daijiao Liu, Fei Tian, Yayue Deng, Jun Chen, Qingjian Lin, Haoyang Zhang, Yuxin Li, Jinglan Gong, Yechang Huang, Liang Zhao, Chengyuan Yao, Hexin Liu, Eng Siong Chng, Xuerui Yang, Gang Yu, Xiangyu Zhang, and Daxin Jiang. Step-audio-r1.5 technical report.arXiv preprint arXiv:2604.25719, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  23. [23]

    GPT-4o System Card

    OpenAI. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  24. [24]

    Qwen3-Omni Technical Report

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. Qwen3-omni technical report.arXiv preprint arXiv:2509.17765, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  25. [25]

    GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot.arXiv preprint arXiv:2412.02612, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  26. [26]

    VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction.arXiv preprint arXiv:2501.01957, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  27. [27]

    Vita-audio: Fast interleaved cross-modal to- ken generation for efficient large speech-language model,

    Zuwei Long, Yunhang Shen, Chaoyou Fu, Heting Gao, Lijiang Li, Peixian Chen, Mengdan Zhang, Hang Shao, Jian Li, Jinlong Peng, Haoyu Cao, Ke Li, Rongrong Ji, and Xing Sun. Vita-audio: Fast interleaved cross-modal token generation for efficient large speech-language model.arXiv preprint arXiv:2505.03739, 2025

    work page arXiv 2025

  28. [28]

    Minicpm-o 2.6: A gemini 2.5 flash level mllm for vision, speech, and full-duplex multimodal live streaming on your phone

    OpenBMB Team. Minicpm-o 2.6: A gemini 2.5 flash level mllm for vision, speech, and full-duplex multimodal live streaming on your phone. https://github.com/OpenBMB/ MiniCPM-o, 2025. Accessed: 2025

    work page 2025

  29. [29]

    Mamba in speech: Towards an alternative to self-attention.IEEE Transactions on Audio, Speech and Language Processing, 2025

    Xiangyu Zhang, Qiquan Zhang, Hexin Liu, Tianyi Xiao, Xinyuan Qian, Beena Ahmed, Eliathamby Ambikairajah, Haizhou Li, and Julien Epps. Mamba in speech: Towards an alternative to self-attention.IEEE Transactions on Audio, Speech and Language Processing, 2025

    work page 2025

  30. [30]

    Code-switching speech recognition under the lens: Model-and data-centric perspectives.IEEE Transactions on Audio, Speech and Language Processing, 2026

    Hexin Liu, Haoyang Zhang, Qiquan Zhang, Xiangyu Zhang, Dongyuan Shi, Eng Siong Chng, and Haizhou Li. Code-switching speech recognition under the lens: Model-and data-centric perspectives.IEEE Transactions on Audio, Speech and Language Processing, 2026

    work page 2026

  31. [31]

    Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation

    Che Liu, Lichao Ma, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, Xuerui Yang, and Fei Tian. Boosting omni-modal language models: Staged post-training with visually debiased evaluation.arXiv preprint arXiv:2605.12034, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  32. [32]

    In2024 IEEE Spo- ken Language Technology Workshop (SLT), pages 1115–1122

    Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H Liu, and Hung-yi Lee. Full-duplex-bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities.arXiv preprint arXiv:2503.04721, 2025. 16 DuplexSLA

    work page arXiv 2025

  33. [33]

    Wildspeech-bench: Benchmarking end-to-end speechllms in the wild,

    Jian Zhang, Linhao Zhang, Bokai Lei, Chuhan Wu, Wei Jia, and Xiao Zhou. Wildspeech-bench: Benchmarking audio llms in natural speech conversation.arXiv preprint arXiv:2506.21875, 2025

    work page arXiv 2025

  34. [34]

    MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark

    Dingdong Wang, Jincenzi Wu, Junan Li, Dongchao Yang, Xueyuan Chen, Tianhua Zhang, and Helen Meng. Mmsu: A massive multi-task spoken language understanding and reasoning benchmark.arXiv preprint arXiv:2506.04779, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark

    S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark.arXiv preprint arXiv:2410.19168, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  36. [36]

    Multi-bench: A multi-turn interactive benchmark for assessing emotional intelligence ability of spoken dialogue models.arXiv preprint arXiv:2511.00850, 2025

    Yayue Deng, Guoqiang Hu, Haiyang Sun, Xiangyu Zhang, Haoyang Zhang, Fei Tian, Xuerui Yang, Gang Yu, and Eng Siong Chng. Multi-bench: A multi-turn interactive benchmark for assessing emotional intelligence ability of spoken dialogue models.arXiv preprint arXiv:2511.00850, 2025

    work page arXiv 2025

  37. [37]

    Talking turns: Benchmarking audio foundation models on turn-taking dynamics.arXiv preprint arXiv:2503.01174, 2025

    Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu, Ruoming Pang, and Shinji Watanabe. Talking turns: Benchmarking audio foundation models on turn-taking dynamics. InThe Thirteenth International Conference on Learning Representations, 2025. arXiv:2503.01174

    work page arXiv 2025

  38. [38]

    VoiceBench: Benchmarking LLM-Based Voice Assistants

    Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. V oicebench: Benchmarking llm-based voice assistants.arXiv preprint arXiv:2410.17196, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  39. [39]

    Air-bench: Benchmarking large audio-language models via generative comprehension

    Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. Air-bench: Benchmarking large audio-language models via generative comprehension. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2402.07729

    work page arXiv 2024

  40. [40]

    Sd-eval: A benchmark dataset for spoken dialogue understand- ing beyond words

    Junyi Ao, Yuancheng Wang, Xiaohai Tian, Dekun Chen, Jun Zhang, Lu Lu, Yuxuan Wang, Haizhou Li, and Zhizheng Wu. Sd-eval: A benchmark dataset for spoken dialogue understand- ing beyond words. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2406.13340

    work page arXiv 2024

  41. [41]

    Chih-Kai Yang, Yu-Kuan Fu, Chen-An Li, Yi-Cheng Lin, Yu-Xiang Lin, Wei-Chih Chen, Ho Lam Chung, Chun-Yi Kuan, Wei-Ping Huang, Ke-Han Lu, and 1 others

    Ruiqi Yan, Xiquan Li, Wenxi Chen, Zhikang Niu, Chen Yang, Ziyang Ma, Kai Yu, and Xie Chen. Uro-bench: Towards comprehensive evaluation for end-to-end spoken dialogue models. InFindings of the Association for Computational Linguistics: EMNLP 2025, 2025. arXiv:2502.17810

    work page arXiv 2025

  42. [42]

    Vocalbench: Benchmarking the vocal conversational abilities for speech interaction models,

    Heyang Liu, Yuhao Wang, Ziyang Cheng, Ronghua Wu, Qunshan Gu, Yanfeng Wang, and Yu Wang. V ocalbench: Benchmarking the vocal conversational abilities for speech interaction models.arXiv preprint arXiv:2505.15727, 2025. 17 DuplexSLA Appendix A Per-Chunk Serialization Case Studies This appendix gives concrete chunk-by-chunk traces of the dual-stream three-c...

    work page arXiv 2025