Pith. sign in

REVIEW 4 major objections 6 minor 16 cited by

FireRedChat claims a pluggable turn-taking controller makes full-duplex voice interaction robust—fewer false interruptions, accurate end-of-turn detection, and latency within 0.3 s of a commercial assistant.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 23:28 UTC pith:5NZERKDH

load-bearing objection Solid engineering contribution to full-duplex voice interaction, held back by missing pVAD enrollment details and thin evaluation. the 4 major comments →

arxiv 2509.06502 v1 pith:5NZERKDH submitted 2025-09-08 cs.SD cs.HC

FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementations

Junjie Chen , Yao Hu , Junjie Li , Kangyue Li , Kun Liu , Wenpeng Li , Xu Li , Ziyuan Li
show 7 more authors
Feiyu Shen Xu Tang Manzhen Wei Yichen Wu Fenglong Xie Kaituo Xu Kun Xie
This is my paper
classification cs.SD cs.HC
keywords full-duplex voice interactionturn-taking controllerpersonalized voice activity detectionend-of-turn detectionsemi-cascaded pipelinebarge-inspeech interactionlow latency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper presents a complete full-duplex voice interaction system that upgrades ordinary half-duplex pipelines into systems where the user can interrupt the agent at any time. Its central claim is that a dedicated turn-taking controller—built from a streaming personalized voice activity detector (pVAD) and a semantic end-of-turn detector—solves the two failure modes of full-duplex interaction: false interruptions from noise or other speakers, and premature responses before the user has finished. The authors implement both a cascaded pipeline (ASR–LLM–TTS) and a semi-cascaded pipeline (an audio LLM that hears the user's speech directly, plus a dialogue-conditioned TTS), and argue the semi-cascaded variant preserves emotional and paralinguistic cues, reduces error propagation, and lowers latency. They propose three system-level metrics—barge-in robustness, end-of-turn detection accuracy, and end-to-end latency—and report that their system suffers fewer false barge-ins, detects semantic ends more accurately, and approaches the latency of a commercial voice assistant. If correct, this makes controllable, natural, real-time full-duplex interaction practical without retraining an end-to-end speech-to-speech model.

Core claim

At the core is the turn-taking controller: a streaming personalized VAD (pVAD) that conditions on a target-speaker embedding from an ECAPA-TDNN encoder, processes 10ms chunks causally, and outputs a speaking probability for the primary speaker only, suppressing interference from noise and other voices. Rather than feeding denoised audio downstream, the system uses pVAD timestamps to slice the original unprocessed audio, preserving acoustic cues for the response model. A fine-tuned BERT classifier serves as the end-of-turn detector, deciding from all accumulated ASR transcripts whether the user has reached a semantic stopping point. On top of this controller, the authors build cascaded (FireR

What carries the argument

The central object is the turn-taking controller, composed of two learned modules. The streaming personalized VAD (pVAD) is a causal convolutional-recurrent network conditioned on a target-speaker embedding, producing a 10ms-resolution speaking probability for the primary speaker; it is what suppresses false barge-ins and enables precise interruption. The end-of-turn detector is a fine-tuned BERT classifier that reads the accumulated transcript and predicts 'continue' or 'stop'—the semantic decision that prevents premature responses. Around these, the semi-cascaded architecture pairs an AudioLLM (which hears speech directly rather than through ASR text) with a dialogue-conditioned TTS, letti

Load-bearing premise

The system assumes a target-speaker embedding is available for the primary user at inference time; the paper does not say how that embedding is enrolled or refreshed in a live session, and the entire barge-in suppression benefit rests on this unstated step.

What would settle it

Run the pVAD in a live two-speaker conversation with an enrolled primary user and measure the false barge-in rate when a second speaker, of the same gender and similar voice, talks at the same loudness; if false barge-ins increase to the level of an unpersonalized VAD, the claimed suppression depends on a degree of speaker discrimination the model may not achieve. A simpler check: measure barge-in success for the primary speaker with and without the target-speaker embedding—if the gap is small, the pVAD's advantage over a plain VAD is not actually due to personalization.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Any half-duplex pipeline—cascaded, semi-cascaded, or end-to-end speech-to-speech—can be upgraded to full duplex by inserting the turn-taking controller, without retraining the pipeline itself.
  • The pVAD's timestamps preserve the original audio for downstream models, so paralinguistic information in the user's voice survives barge-in handling.
  • The semi-cascaded configuration replaces ASR+LLM with a single AudioLLM, reducing error propagation and simplifying the chain while keeping TTS separately optimizable.
  • The three proposed metrics (barge-in, EoT accuracy, end-to-end latency) give voice-interaction systems a common yardstick for naturalness, control accuracy, and efficiency.
  • At the reported T90 of 170 ms with a false barge-in rate of 10.2%, the system shows that suppressing spurious interruptions costs only about 30 ms of extra response delay compared with a more trigger-happy baseline.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's implicit promise is that pVAD's speaker conditioning can be deployed in any multi-speaker setting; a natural extension would be to test whether enrollment from a short sample (e.g., a user's first sentence) keeps false barge-in low over a long session as the background and the user's voice drift.
  • Because the EoT detector works on transcripts, it should transfer to any ASR output; one testable extension is whether combining acoustic pause features with the semantic classifier further reduces 'continue' errors on incomplete but paused utterances.
  • The same turn-taking controller could be applied to an end-to-end speech-to-speech model, which the paper mentions as possible but does not implement; the metrics proposed here would allow a direct comparison of controllability versus end-to-end latency.
  • The 0.3 s latency gap to the commercial system is attributed to missing streaming ASR, streaming TTS input, and chunk size; these are engineering knobs, suggesting the gap is closable rather than architectural.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents FireRedChat, a modular full-duplex voice interaction system centered on a turn-taking controller that combines a streaming personalized voice activity detector (pVAD) with a semantic end-of-turn (EoT) detector. The controller is designed to upgrade half-duplex pipelines (cascaded, semi-cascaded, or speech-to-speech) to full duplex. The authors instantiate cascaded (FireRedASR, Qwen2.5, FireRedTTS-1s) and semi-cascaded (AudioLLM, FireRedTTS-2) variants, and propose three system-level metrics: barge-in, EoT detection accuracy, and end-to-end latency. Experiments against LiveKit, Ten, and DouBao report lower false barge-in rates, competitive EoT accuracy, and latency approaching an industrial system. The central claim is that pVAD and the semantic EoT detector enable more robust, natural, real-time full-duplex interaction than existing open frameworks.

Significance. If the claims hold, the paper makes a useful practical contribution: a pluggable, internally consistent full-duplex stack with open-sourced ASR/TTS components, a concrete set of system-level evaluation metrics, and a design that allows heterogeneous half-duplex pipelines to be upgraded to full duplex. The proposed pVAD idea is appealing because it addresses a real problem—false barge-ins from noise and non-primary speakers—and the EoT detector is a sensible semantic complement to acoustic VAD. The paper also reports reproducible artifacts (demos, FireRedASR, FireRedTTS) and is honest about standardizing the LLM across frameworks. However, several load-bearing experimental and methodological gaps currently prevent the strong conclusions from being fully supported: the enrollment mechanism for the personalized VAD is unspecified, the semi-cascaded advantages are not evaluated at all, and the quantitative results lack variance estimates and are based on small synthetic test sets.

major comments (4)
  1. [§2.2.1, §3.1] The pVAD is conditioned on a target-speaker ECAPA-TDNN embedding, but the manuscript never states how this embedding is obtained at inference time. A live full-duplex session must know which speaker is primary before suppressing noise/non-primary speakers; no enrollment procedure, adaptation, or fallback is given. Section 3.1 also does not state how the embedding was generated for the 1,000 test utterances per language. If the embedding is extracted from the same utterance that is later tested (an oracle condition), the reported 10.2% false barge-in rate and T90=170 ms are not reproducible in practice. Since pVAD is the main differentiator in the barge-in comparison, this is load-bearing for the headline claim of fewer false interruptions. Please specify the enrollment protocol (e.g., short enrollment utterance, speaker diarization, or voiceprint database) and report barge-in results und
  2. [§2.3.2, §3] Several central claims about the semi-cascaded implementation are not evaluated. The abstract and §2.3.2 state that the semi-cascaded variant 'captures emotional and paralinguistic cues, yields more coherent responses, lowers latency and error propagation, and improves robustness.' However, Table 1 and all experiments use the cascaded configuration (FireRedASR+Qwen2.5+FireRedTTS-1s); no experiment compares cascaded vs. semi-cascaded on paralinguistic awareness, response coherence, error propagation, or latency. The latency results in Table 4 cannot be attributed to the semi-cascaded design. Either add a head-to-head evaluation of the two implementations (e.g., emotion/paralinguistic metrics, human coherence ratings, ASR-error injection, per-stage latency), or restrict the claims to the cascaded system and present the semi-cascaded advantages as design hypotheses.
  3. [§3.1, §3.3] The quantitative evidence consists of single point estimates without variance. Barge-in T90 and false barge-in rates in Table 2, EoT accuracies in Table 3, and P50/P95 latencies in Table 4 are each reported once with no confidence intervals, number of repeated trials, or significance tests. Latency is based on 25 synthetic utterances only (Table 4) and measured via screen recording of a phone, which introduces uncontrolled jitter. With n=25 and no repeated runs, a 0.3 s advantage over DouBao is not established. Please report means/standard deviations or confidence intervals over multiple runs and, where possible, larger and more diverse test sets, including real conversational data.
  4. [§3.3] The latency comparison is not apples-to-apples. Table 1 states that the LLM was standardized to Qwen2.5 for LiveKit, Ten, and FireRedChat, but DouBao is a closed commercial system accessed via mobile app; its ASR, LLM, TTS, and turn-taking stack are unknown. Differences could come from model size, streaming capability, network, or device rather than the compared architectures. The claim that FireRedChat 'narrows the interaction-latency gap with industrial-grade systems' is therefore only weakly supported. I recommend either limiting the comparison with DouBao to an anecdotal reference, or adding a decomposition of latency (VAD, ASR, LLM, TTS, network) and a comparable open industrial baseline.
minor comments (6)
  1. [§1] Typo: 'two models in paralle' should be 'parallel'. Also, the text repeatedly renders VAD with a spurious space ('V AD', 'pV AD'); please fix throughout.
  2. [§3.1] The sentence 'we constructed 1,000 utterances each in Chinese and English' is ambiguous: does this mean 1,000 per language or 1,000 total split across languages? Clarify, and state whether the barge-in metrics are averaged over both languages or reported separately.
  3. [Table 1] The table is labeled 'Configurations between different systems' but only lists component names. To be reproducible, please include model versions, checkpoints, and any hyperparameters that affect latency or accuracy (e.g., chunk sizes, streaming settings).
  4. [§2.2.1] The pVAD description gives high-level architecture (mel features, causal conv, ECAPA-TDNN embedding, GRU) but no model size, input frame size, GRU hidden units, or training schedule. Provide enough detail to reimplement or, if the model is proprietary, offer a reference implementation or pseudocode.
  5. [§3.3] The 25 'representative synthetic utterances' are not described in terms of selection criteria, duration distribution, or emotional/pause coverage. A supplementary table listing the utterances or a description of the sampling procedure would make the latency experiment more interpretable.
  6. [§3.2, References] Reference [40] (TEN_Turn_Detection) lacks a URL, version, or access details. If the test set is publicly available, provide a link; if not, describe how it can be obtained for reproducibility.

Circularity Check

0 steps flagged

No significant circularity: system claims rest on direct measurement against external frameworks and benchmarks; self-citations are component references only.

full rationale

The paper's central claims are empirical system-performance claims: fewer false interruptions, more accurate semantic end-of-turn detection, and lower latency. These are supported by direct measurements in Tables 2-4 against external systems (LiveKit, Ten, DouBao) and the external TEN_Turn_Detection benchmark [40]. The pVAD and EoT modules are trained on purpose-built data and evaluated on held-out synthetic mixtures and an external test set; no fitted parameter is renamed as a prediction, and no equation in the paper reduces one claimed quantity to another by construction. Self-citations (FireRedASR [2], FireRedTTS-1s [10], FireRedTTS-2 [14]) are used as component references in the implemented pipelines; they do not carry the argument for the headline conclusions, which are measured de novo. The main weakness is an unspecified enrollment procedure for the target-speaker embedding in pVAD (Section 2.2.1), which could affect real-world reproducibility of the barge-in results; this is a correctness/reproducibility concern, not circularity, because the barge-in metric is a behavioral measurement rather than a quantity derived from the embedding. No load-bearing uniqueness theorem, ansatz-by-citation, or self-definitional reduction is present.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claims rest primarily on the pVAD module (whose architecture and training are partially described but whose inference-time speaker enrollment is unstated), the EoT classifier (trained on an unreleased constructed corpus), and the latency measurement protocol. No fundamentally new physical entities are introduced.

free parameters (4)
  • pVAD mixture SNR range = 0-30 dB
    Training mixtures use SNR randomly sampled from 0-30 dB for both target-interferer and target-noise; this design choice affects barge-in behavior and is not derived from external constraints.
  • Interference type probability = 50%
    Interfering speakers and noise are injected with equal probability in pVAD training; this choice shapes the model's balance between suppressing noise and suppressing other speakers.
  • T90 threshold = 90%
    Barge-in success is reported as the latency to reach 90% accuracy; the threshold is chosen by the authors and directly used to compare systems.
  • EoT training corpus size = 830k
    The EoT classifier is trained on 830k constructed text instances; the scale and construction method are choices that affect the reported accuracy.
axioms (4)
  • domain assumption The target-speaker embedding is available at inference for pVAD without describing an enrollment procedure.
    Section 2.2.1 describes conditioning on an ECAPA-TDNN embedding but does not state how the primary speaker's embedding is obtained at test time. The barge-in filtering depends on this.
  • domain assumption FireRedASR is robust enough to noise and overlapping speech that using original audio (not pVAD-denoised) is safe.
    Section 2.1 explicitly relies on this property to justify feeding non-denoised segments to the ASR; if ASR were not robust, barge-in segments could be mis-transcribed.
  • domain assumption Partial spans sampled from complete utterances are a valid proxy for unfinished turns in EoT training.
    Section 2.2.2 constructs the EoT training set this way; the validity of this proxy for real user pauses is assumed without external validation.
  • domain assumption Screen-recording with fixed phone position and volume yields a fair latency comparison across systems.
    Section 3.3 relies on this measurement setup; network conditions and playback hardware are otherwise uncontrolled, so the latency numbers are setup-dependent.

pith-pipeline@v1.3.0-alltime-deepseek · 10307 in / 11371 out tokens · 114003 ms · 2026-08-04T23:28:52.238701+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementations." pith.science (2026). https://pith.science/paper/5NZERKDH

@misc{pith2026250906502,
  author       = {Pith},
  title        = {Pith review of: FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5NZERKDH}},
  note         = {Machine review of arXiv:2509.06502}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Full-duplex voice interaction allows users and agents to speak simultaneously with controllable barge-in, enabling lifelike assistants and customer service. Existing solutions are either end-to-end, difficult to design and hard to control, or modular pipelines governed by turn-taking controllers that ease upgrades and per-module optimization; however, prior modular frameworks depend on non-open components and external providers, limiting holistic optimization. In this work, we present a complete, practical full-duplex voice interaction system comprising a turn-taking controller, an interaction module, and a dialogue manager. The controller integrates streaming personalized VAD (pVAD) to suppress false barge-ins from noise and non-primary speakers, precisely timestamp primary-speaker segments, and explicitly enable primary-speaker barge-ins; a semantic end-of-turn detector improves stop decisions. It upgrades heterogeneous half-duplex pipelines, cascaded, semi-cascaded, and speech-to-speech, to full duplex. Using internal models, we implement cascaded and semi-cascaded variants; the semi-cascaded one captures emotional and paralinguistic cues, yields more coherent responses, lowers latency and error propagation, and improves robustness. A dialogue manager extends capabilities via tool invocation and context management. We also propose three system-level metrics, barge-in, end-of-turn detection accuracy, and end-to-end latency, to assess naturalness, control accuracy, and efficiency. Experiments show fewer false interruptions, more accurate semantic ends, and lower latency approaching industrial systems, enabling robust, natural, real-time full-duplex interaction. Demos: https://fireredteam.github.io/demos/firered_chat.

Figures

Figures reproduced from arXiv: 2509.06502 by Feiyu Shen, Fenglong Xie, Junjie Chen, Junjie Li, Kaituo Xu, Kangyue Li, Kun Liu, Kun Xie, Manzhen Wei, Wenpeng Li, Xu Li, Xu Tang, Yao Hu, Yichen Wu, Ziyuan Li.

Figure 1
Figure 1. Figure 1: FireRedChat System Modules. controller delivers human-like full-duplex interactions: the streaming personalized voice activity de￾tection (pVAD) suppresses background noise and competing speakers to reduce false barge-ins, more precisely detects intentional interruptions initiated by the primary speaker, and produces accurate timestamps for the primary speaker’s speech segments. In addition, an End-of-Turn… view at source ↗
Figure 2
Figure 2. Figure 2: FireRedChat Voice Interaction Flow. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Survey of Full-Duplex Spoken Dialogue Systems: Architectural Hierarchy, Interaction Ontology, and Decision State Machine

    eess.AS 2026-06 accept novelty 7.0

    A survey proposing an L0-L3 architectural hierarchy, T×I×R interaction ontology, and IDLE/LISTEN/SPEAK/WAIT/DUAL decision state machine for full-duplex spoken dialogue systems, documenting a realization gap between ar...

  2. PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects

    cs.CL 2026-05 unverdicted novelty 7.0

    PolySpeech-100 is a new benchmark for native-level speech comprehension across 110 linguistic variants that evaluates 22 models and reports E2E advantages on dialects, robustness gaps on low-resource languages, and de...

  3. Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0

    RoleJudge is a multidimensional evaluation framework for speech-character alignment in audio LLMs, backed by the RoleChat dataset and multi-stage RL training with standard alignment to reduce reward issues.

  4. Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

    cs.CL 2026-07 conditional novelty 6.0

    Lychee-FD resolves modality interference in full-duplex spoken language models by separating acoustic and semantic parameters in deep layers and adding a dense semantic alignment channel, achieving state-of-the-art pe...

  5. Next-Turn: Duration-Aware Streaming Endpoint Detection via Time-to-Next-Speech-Onset Prediction

    cs.SD 2026-06 unverdicted novelty 6.0

    Next-Turn introduces time-to-next-speech-onset prediction for duration-aware streaming endpoint detection, reporting a 25.9% improvement in accuracy within 320 ms.

  6. UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction

    cs.AI 2026-04 unverdicted novelty 6.0

    UAF is the first unified audio front-end LLM that turns multiple front-end tasks into one sequence prediction model processing streaming audio chunks and reference prompts to output semantic and control tokens for ful...

  7. FastTurn: Unifying Acoustic and Streaming Semantic Cues for Low-Latency and Robust Turn Detection

    cs.SD 2026-04 unverdicted novelty 6.0

    FastTurn unifies acoustic features and streaming CTC decoding for low-latency, robust turn detection in full-duplex dialogue systems and releases a realistic human-dialogue test set.

  8. X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment

    cs.LG 2026-07 conditional novelty 5.0

    X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.

  9. DuplexOmni: Real-Time Listening, Seeing, Thinking, and Speaking for Full-Duplex Interaction

    cs.HC 2026-06 unverdicted novelty 5.0

    DuplexOmni achieves real-time full-duplex multimodal interaction by separating an interaction layer from a pluggable thinking layer, supported by a Writer-Director pipeline for continuous-interaction training data.

  10. A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook

    cs.SD 2026-05 unverdicted novelty 5.0

    A survey of Large Audio Language Models that establishes a taxonomy of trustworthiness vulnerabilities and proposes a Defense-in-Depth roadmap for audio intelligence.

  11. Full-Duplex Interaction in Spoken Dialogue Systems: A Comprehensive Study from the ICASSP 2026 HumDial Challenge

    eess.AS 2026-04 unverdicted novelty 5.0

    A new HumDial-FDBench benchmark and real human-recorded dual-channel dataset are released to assess full-duplex dialogue systems on interruptions and conversational flow.

  12. FastTurn: Unifying Acoustic and Streaming Semantic Cues for Low-Latency and Robust Turn Detection

    cs.SD 2026-04 unverdicted novelty 5.0

    FastTurn fuses streaming CTC semantics with acoustic cues for lower-latency, more accurate turn detection in full-duplex dialogue and releases a real-dialogue test set.

  13. FastTurn: Unifying Acoustic and Streaming Semantic Cues for Low-Latency and Robust Turn Detection

    cs.SD 2026-04 unverdicted novelty 5.0

    FastTurn fuses streaming CTC semantics with acoustic cues for lower-latency, more accurate turn detection and ships a real-dialogue test set.

  14. Sharp spectral estimates for free boundary problems arising in plasma physics

    math.AP 2026-04 unverdicted novelty 5.0

    For a constrained superlinear free-boundary plasma model, the non-local first eigenvalue σ₁ is always positive on balls in every dimension N≥2, despite lacking a general Faber–Krahn property.

  15. IRAF: Interference-Resilient Adaptive Fusion for Noise-Robust End-to-End Full-Duplex Spoken Dialogue Systems

    cs.SD 2026-06 unverdicted novelty 4.0

    IRAF introduces an adaptive fusion module that uses a predicted scalar reliability gate to reduce the impact of interfering speakers on user audio representations in end-to-end full-duplex spoken dialogue systems, wit...

  16. Toward Native Multimodal Modeling: A Roadmap

    cs.CV 2026-05 unverdicted novelty 3.0

    A roadmap that defines architectural nativity for multimodal models and categorizes them into Multi-to-Text, Multi-to-Target, and Multi-to-Multi types while outlining an industrial pipeline toward unified transformer-...

Reference graph

Works this paper leans on

40 extracted references · 8 canonical work pages · cited by 14 Pith papers · 1 internal anchor

  1. [1]

    Language model can listen while speaking

    Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, and Xie Chen. Language model can listen while speaking. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24831–24839, 2025

    2025

  2. [2]

    Fireredasr: Open-source industrial-grade mandarin speech recognition models from encoder-decoder to llm integration.arXiv preprint arXiv:2501.14350, 2025

    Kai-Tuo Xu, Feng-Long Xie, Xu Tang, and Yao Hu. Fireredasr: Open-source industrial-grade mandarin speech recognition models from encoder-decoder to llm integration.arXiv preprint arXiv:2501.14350, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InInternational conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  4. [4]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. 9

    Pith/arXiv arXiv 2023

  6. [6]

    Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1– 113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1– 113, 2023

    work page 2023

  7. [7]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    Qwen Team. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications.arXiv preprint arXiv:2409.03283, 2024

    Hao-Han Guo, Yao Hu, Kun Liu, Fei-Yu Shen, Xu Tang, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications.arXiv preprint arXiv:2409.03283, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Fireredtts-1s: An upgraded streamable foundation text-to-speech system.arXiv preprint arXiv:2503.20499, 2025

    Hao-Han Guo, Yao Hu, Fei-Yu Shen, Xu Tang, Yi-Chen Wu, Feng-Long Xie, and Kun Xie. Fireredtts-1s: An upgraded streamable foundation text-to-speech system.arXiv preprint arXiv:2503.20499, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens.arXiv preprint arXiv:2407.05407, 2024

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens.arXiv preprint arXiv:2407.05407, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training.arXiv preprint arXiv:2505.17589, 2025

    Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, et al. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training.arXiv preprint arXiv:2505.17589, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117, 2024

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Fireredtts-2: Towards long conversational speech generation for podcast and chatbot.arXiv preprint arXiv:2509.02020, 2025

    Kun Xie, Feiyu Shen, Junjie Li, Fenglong Xie, Xu Tang, and Yao Hu. Fireredtts-2: Towards long conversational speech generation for podcast and chatbot.arXiv preprint arXiv:2509.02020, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Qwen2-audio technical report.arXiv preprint arXiv:2407.10759, 2024

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuan- jun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report.arXiv preprint arXiv:2407.10759, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Salmonn: Towards generic hearing abilities for large language models.arXiv preprint arXiv:2310.13289, 2023

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models.arXiv preprint arXiv:2310.13289, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    On decoder-only architecture for speech-to-text and large language model integration

    Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, et al. On decoder-only architecture for speech-to-text and large language model integration. In2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. IEEE, 2023

    work page 2023

  19. [19]

    Midashenglm: Efficient audio understanding with general audio captions

    Horizon Team, MiLM Plus. Midashenglm: Efficient audio understanding with general audio captions. Technical report, Xiaomi Inc., 2025. Contributors: Heinrich Dinkel et al. (listed alphabetically in Appendix B)

    work page 2025

  20. [20]

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.arXiv preprint arXiv:2305.11000, 2023

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.arXiv preprint arXiv:2305.11000, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Lauragpt: Listen, attend, understand, and regenerate audio with gpt.arXiv preprint arXiv:2310.04673, 2023

    Zhihao Du, Jiaming Wang, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, et al. Lauragpt: Listen, attend, understand, and regenerate audio with gpt.arXiv preprint arXiv:2310.04673, 2023. 10

    Pith/arXiv arXiv 2023

  22. [22]

    Mini-omni: Language models can hear, talk while thinking in streaming.arXiv preprint arXiv:2408.16725, 2024

    Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming.arXiv preprint arXiv:2408.16725, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Llama-omni: Seamless speech interaction with large language models.arXiv preprint arXiv:2409.06666, 2024

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models.arXiv preprint arXiv:2409.06666, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025

    Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Step-audio: Unified understanding and generation in intelligent speech interaction.arXiv preprint arXiv:2502.11946, 2025

    Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction.arXiv preprint arXiv:2502.11946, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Step-audio 2 technical report.arXiv preprint arXiv:2507.16632, 2025

    Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, et al. Step-audio 2 technical report.arXiv preprint arXiv:2507.16632, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report.arXiv preprint arXiv:2503.20215, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Generative spoken dialogue language modeling.Transactions of the Association for Computational Linguistics, 11:250–266, 2023

    Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Generative spoken dialogue language modeling.Transactions of the Association for Computational Linguistics, 11:250–266, 2023

    2023

  29. [29]

    Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm.arXiv preprint arXiv:2411.00774, 2024

    Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm.arXiv preprint arXiv:2411.00774, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Minmo: A multimodal large language model for seamless voice interaction.arXiv preprint arXiv:2501.06282, 2025

    Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, et al. Minmo: A multimodal large language model for seamless voice interaction.arXiv preprint arXiv:2501.06282, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog

    Erik Ekstedt and Gabriel Skantze. Turngpt: a transformer-based language model for predicting turn-taking in spoken dialog.arXiv preprint arXiv:2010.10874, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  33. [33]

    Vita: Towards open-source interactive omni multimodal llm.arXiv preprint arXiv:2408.05211, 2024

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Yuhang Dai, Meng Zhao, Yi-Fan Zhang, Shaoqi Dong, Yangze Li, Xiong Wang, et al. Vita: Towards open-source interactive omni multimodal llm.arXiv preprint arXiv:2408.05211, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Flexduo: A pluggable system for enabling full-duplex capabilities in speech dialogue systems

    Borui Liao, Yulong Xu, Jiao Ou, Kaiyuan Yang, Weihua Jian, Pengfei Wan, and Di Zhang. Flexduo: A pluggable system for enabling full-duplex capabilities in speech dialogue systems. arXiv preprint arXiv:2502.13472, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Two-dimensional convo- lutional recurrent neural networks for speech activity detection

    Anastasios Vafeiadis, Eleftherios Fanioudakis, Ilyas Potamitis, Konstantinos V otis, Dimitrios Giakoumis, Dimitrios Tzovaras, Liming Chen, and Raouf Hamzaoui. Two-dimensional convo- lutional recurrent neural networks for speech activity detection. International Speech Communi- cation Association, 2019

    work page 2019

  36. [36]

    Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification.arXiv preprint arXiv:2005.07143, 2020

    Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification.arXiv preprint arXiv:2005.07143, 2020

    Pith/arXiv arXiv 2005

  37. [37]

    Gate-variants of gated recurrent unit (gru) neural networks

    Rahul Dey and Fathi M Salem. Gate-variants of gated recurrent unit (gru) neural networks. In 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS), pages 1597–1600. IEEE, 2017. 11

    work page 2017

  38. [38]

    Musan: A music, speech, and noise corpus

    David Snyder, Guoguo Chen, and Daniel Povey. Musan: A music, speech, and noise corpus. arXiv preprint arXiv:1510.08484, 2015

    Pith/arXiv arXiv 2015

  39. [39]

    Icassp 2023 deep noise suppression challenge.IEEE Open Journal of Signal Processing, 5:725–737, 2024

    Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cutler, Alex Ju, Mehdi Zohourian, Min Tang, Mehrsa Golestaneh, et al. Icassp 2023 deep noise suppression challenge.IEEE Open Journal of Signal Processing, 5:725–737, 2024

    work page 2023

  40. [40]

    Ten turn detection: Turn detection for full-duplex dialogue communication, 2025

    TEN Team. Ten turn detection: Turn detection for full-duplex dialogue communication, 2025. 12

    work page 2025