Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A duplex speech-to-speech model can be built without speech-text pretraining by fusing user speech embeddings with parallel agent text and codec channels, and it beats prior duplex models on reasoning, turn-taking, and barge-in.

desk verdict A genuinelynew duplex S2S architecture without speech-text pretraining, but the turn-taking evaluation leans on a fixed 0.64 s silence and a single baseline; worth refereeing with revisions. read the letter →

arxiv 2505.15670 v4 pith:AOFTEVK7 submitted 2025-05-21 cs.CL cs.SDeess.AS

Ke Hu , Ehsan Hosseini-Asl , Chen Chen , Edresson Casanova , Subhankar Ghosh , Piotr Żelasko , Zhehuai Chen , Jason Li
show 2 more authors
Jagadeesh Balam Boris Ginsburg
This is my paper · ORCID
classification cs.CLcs.SDeess.AS
keywords duplexspeech-to-speechstreamingspeechencoderbarge-inturn-takingcodecpersonalizationchannelfusionmulti-channelnext-tokenpredictionspokendialogue
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that full-duplex spoken dialogue does not require a separate speech-text pretraining stage: a pretrained streaming speech encoder can bring user audio into an ordinary text LLM, while agent speech and text are predicted in parallel as separate channels. This lowers the data and compute barrier enough that a 1.1B text model becomes a duplex voice agent, and it lets the agent voice be improved independently by fine-tuning the codec on a target speaker. The paper reports that this model outperforms the previous duplex baseline on reasoning, turn-taking, and barge-in metrics, and is competitive with an ideal cascaded text pipeline. It also makes training and inference code publicly available.

What carries the argument

The central object is channel fusion of two simultaneously active streams in a decoder-only LLM: user embeddings from a pretrained 100M streaming encoder at 80 ms frames are summed with agent token embeddings made of text tokens and four parallel 12.5 Hz codec codebooks. A one-token delay on the speech channels lets text lead slightly. The codec's four independent codebooks, built with Finite Scalar Quantization, are predicted in parallel at each timestep, and the whole model is trained by multi-channel next-token prediction with text loss weighted 3 and speech loss weighted 1. Turn-level alignment of text and speech with separate start and end tokens and silence padding is what turns ordinary QA and conversation data into a duplex training format.

What would settle it

Measure barge-in success and false-alarm rates in live conversations where real users pause with natural variability instead of the fixed 0.64-second training silence; if false alarms rise sharply or success drops well below the reported 94.5%, the claimed turn-taking advantage is an artifact of the synthetic silence.

Watch

Extended reading notes

Core claim

The central claim is that duplex conversation can be modeled directly: continuous user speech is encoded by a pretrained streaming encoder, and those embeddings are summed with parallel agent text and codec-speech token embeddings inside a causal LLM, so the model predicts what the agent says and how it sounds at the same time. Agent speech is tokenized with a 0.6 kbps codec that uses four independent parallel codebooks, and text and speech channels are aligned at the turn level with their own start and end tokens. Because the agent stream and user stream are handled by separate modules, the codec can be fine-tuned on a target voice without retraining the dialogue model. The paper's results show higher barge-in success, lower barge-in latency, equal false-alarm rate, and better reasoning scores than the previous duplex baseline, while using a smaller backbone and no speech-text pretraining.

Load-bearing premise

The learned turn-taking and barge-in behavior depends on a fixed 0.64-second silence inserted between user and agent turns in all training data, so if real user pause patterns differ from this synthetic constant, the reported barge-in and false-alarm trade-off may not transfer to real conversations.

Editorial extensions

If this is right

  • Any text LLM can be turned into a duplex voice agent by fine-tuning with a streaming encoder and codec speech tokens, since no speech-text pretraining step is required.
  • Agent voice quality is decoupled from dialogue reasoning: fine-tuning the codec on a target speaker improves both audio reconstruction quality and the word-overlap score computed from transcripts of generated speech, even at half the bitrate of prior codecs.
  • The barge-in behavior transfers from synthetic silence-based data to evaluation: on an impatient-user set the model stops speaking within 1.5 seconds in 94.5% of interruptions, compared with 55.1% for the prior duplex baseline.
  • Reasoning quality is competitive with an optimal cascaded system that feeds ground-truth user text into an LLM, while avoiding the latency and pipeline complexity of cascades.
  • Open training and inference code means the recipe can be reproduced and adapted by others without rebuilding the data pipeline from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, the practical cost of building a duplex voice assistant shifts almost entirely to data curation and codec voice selection, because the model architecture reduces to fine-tuning an existing text LLM.
  • One testable extension is to draw turn-taking silence durations from real conversation corpora rather than the fixed 0.64-second value, which could improve transfer to users with atypical pause patterns.
  • Because user and agent are modeled as separate summed channels, the same fusion mechanism could be extended to multi-speaker or multi-agent dialogue by adding more channels, not by changing the learning objective.
  • A direct ablation would replace the pretrained streaming encoder with a randomly initialized one; if performance collapses, the 'no speech pretraining' claim is really carried by the encoder's pretrained speech understanding.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SALM-Duplex, a duplex speech-to-speech model that combines a streaming speech encoder for continuous user input with a decoder-only LLM that predicts text and low-bitrate codec speech tokens in parallel. The authors train on synthetic TTS data with multi-turn and barge-in examples, using TinyLlama-1.1B as the backbone and a 0.6 kbps personalized NanoCodec for agent speech. They compare against Moshi on barge-in performance, first-response latency, UTMOS, and GPT-based reasoning scores, and report better results on most of these metrics. The paper also releases training and inference code, and claims to be the first duplex S2S model that does not require speech-text pretraining.

Significance. If the results hold, the architecture is a meaningful simplification: a duplex conversational agent can be built by fine-tuning an ordinary text LLM with a streaming encoder and a low-bitrate codec, without a dedicated speech-text pretraining stage. The codec personalization result, showing that a 0.6 kbps personalized codec can outperform 1.1--1.2 kbps codecs on reconstruction metrics for the target speaker, is a useful contribution. The open-source release is a concrete strength for reproducibility. However, the empirical support for the behavioral claims is currently limited by the fixed synthetic timing used in both training and evaluation, the reliance on a single baseline, and the use of automated metrics without uncertainty quantification.

major comments (3)
  1. [Secs. 4.1, 4.2, and Table 2] The barge-in and turn-taking evaluation is confounded by a fixed training-time timing constant. Sec. 4.1 states that 'we insert a 0.64s silence between user and agent before the agent speaks', and Sec. 4.2 keeps exactly 0.64s of agent speech before a barge-in cutoff. The Impatient set is then created by halving the same synthetic silence rather than sampling from a distribution of natural pauses or interruption offsets. The reported barge-in latency of 0.69s in Table 2 is within 0.05s of this training constant, which suggests the model may have learned a fixed-delay policy rather than a general turn-taking rule. The claimed superiority over Moshi on turn-taking and barge-in is therefore not established for natural conversations. Please evaluate with varied silence durations, real human pause distributions, and barge-in onsets, and report barge-in success and false-alarm rates as a function of interruption offset.
  2. [Sec. 6.2 and Table 3] The reasoning comparison uses one GPT-based score computed on ASR transcripts, one held-out shard per dataset, no error bars or significance tests, and only Moshi as a duplex baseline. The abstract's claim that the model outperforms previous duplex models in reasoning is broader than this evidence. Please report confidence intervals across shards or bootstrap samples, and ideally add at least one additional open duplex baseline; a secondary human or reference-based metric would also strengthen the claim.
  3. [Secs. 5.2 and 6.1] The turn-taking metrics are not symmetric across the two systems. The false-alarm definition excludes user stops within 0.1s, and Moshi's first-response latency is listed as n/a because the authors state that Moshi almost always responds before the user finishes talking. Because the two systems operate under different turn-taking policies, comparing barge-in success rates at a fixed cutoff may be misleading. Please make the metric definitions symmetric and report the distribution of interruption timings, or use a common decision threshold for both models.
minor comments (5)
  1. [Sec. 2] The phrase 'speech-texet data' appears to be a typo for 'speech-text data'.
  2. [Sec. 5.2] The evaluation uses one shard per dataset; please specify the number of conversations or utterances in each shard so readers can judge the size of the evaluation.
  3. [Table 3] The 'GT+LLM' baseline is labeled 'optimal', but it uses ground-truth user-turn texts; the label should be qualified as 'text-oracle' to avoid implying that it is an upper bound on all possible cascaded systems.
  4. [Sec. 6.1 and Table 2] The statement that Moshi 'almost always responds before the user finishes talking' should appear with the Table 2 caption, since it explains the n/a entry for Moshi's first-response latency.
  5. [Sec. 6.3 and Table 4] Table 4 reports both audio-reconstruction metrics and S2S ASR-BLEU values; please clarify which evaluation set is used for each column, since the reconstruction metrics are described on 228 test samples while ASR-BLEU appears to be computed on the ASR-QA set.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central S2S model is trained end-to-end and compared against an external baseline; self-citations are implementation references, not load-bearing derivations.

full rationale

The paper's derivation chain is a direct architecture-plus-training recipe: a pretrained streaming encoder encodes user speech, a codec tokenizes agent speech, and a text LLM is fine-tuned to predict text and codec channels jointly. Nothing in this chain is defined in terms of the reported outcomes. The claim 'first duplex S2S model without requiring speech pretrain' rests on the design choice of using a pretrained encoder as an input rather than pretraining the LLM on speech-text pairs; this is a structural fact about the training pipeline, not a renamed version of the evaluation results. Self-citations (NeMo [32], SALM [7], instruction-data generation [29]) are used as tooling and data-generation precedents; none is invoked as a uniqueness theorem or as the justification for the central claim. The one potentially circular-looking element is the fixed 0.64 s silence inserted after user turns (Sec. 4.1) and kept as 0.64 s of agent audio after barge-in cutoffs (Sec. 4.2), which the paper explicitly discloses in Sec. 6.1 as affecting 1st response latency. This is a training-data constant, not a fitted parameter, and the reported barge-in latency and success rates are measurements of the trained model rather than quantities algebraically forced by that constant. The concern that real user pause variability may not transfer is an external-validity limitation, not a circularity; the paper itself flags the latency artifact. Therefore no circular step meets the required evidentiary standard.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several hand-set data generation and training choices (silence duration, loss weighting, channel delay) and on assumptions inherited from the streaming encoder and NanoCodec. No new physical or conceptual entities are introduced. The free parameters are not fitted to maximize the reported metrics, but they directly shape the turn-taking behavior that the paper evaluates.

free parameters (6)
  • silence_duration = 0.64s
    Hand-chosen pause inserted before agent speech in training data to suppress unexpected barge-in; directly sets turn-taking latency and false alarm trade-off.
  • text_loss_weight = 3
    Weight for text channel loss relative to speech channel loss, set in Sec. 5.1.
  • speech_loss_weight = 1
    Weight for speech channel loss relative to text channel, set in Sec. 5.1.
  • speech_channel_delay = 1 token
    One-token delay on speech channels to condition on text context, introduced in Sec. 3.1.
  • barge_in_success_window = 1.5s
    Evaluation threshold for whether the agent stopped in time; affects reported success rate in Sec. 5.2.
  • false_alarm_grace_period = 0.1s
    User stops within 0.1s are not counted as false alarms; the authors note this favors Moshi's proactive behavior, changing the false alarm comparison.
assumptions (4)
  • domain assumption Turn-level text-speech alignment is sufficient for learning the agent text-speech mapping.
    The paper says word-level alignment was tried and showed no improvement (Sec. 3.1), but turn-level alignment is still a modeling choice that could underpin text-speech coherence.
  • domain assumption The pretrained streaming encoder embeddings can be summed with codec embeddings without a cross-modal attention layer.
    Channel fusion via summation in Sec. 3 assumes the two embedding spaces are compatible enough for the LLM to interpret.
  • domain assumption NanoCodec's independent parallel codebooks allow all four channels to be predicted simultaneously without inter-codebook autoregression.
    Borrowed from NanoCodec [23]; if the independence assumption fails, parallel prediction would degrade speech quality.
  • domain assumption A fixed speaker for agent TTS training data is sufficient, with codec personalization, for high quality agent voices.
    Training uses a fixed agent voice (Sec. 4.1) and personalizes the codec on 21k utterances of the same target speaker; generalization to other voices is not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model." pith.science (2026). https://pith.science/paper/AOFTEVK7

@misc{pith2026250515670,
  author       = {Pith},
  title        = {Pith review of: SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AOFTEVK7}},
  note         = {Machine review of arXiv:2505.15670}
}
read the original abstract

Spoken dialogue is an intuitive form of human-computer interaction, yet current speech language models often remain constrained to turn-based exchanges, lacking real-time adaptability such as user barge-in. We propose a novel duplex speech to speech (S2S) architecture featuring continuous user inputs and codec agent outputs with channel fusion that directly models simultaneous user and agent streams. Using a pretrained streaming encoder for user input enables the first duplex S2S model without requiring speech pretrain. Separate architectures for agent and user modeling facilitate codec fine-tuning for better agent voices and halve the bitrate (0.6 kbps) compared to previous works. Experimental results show that the proposed model outperforms previous duplex models in reasoning, turn-taking, and barge-in abilities. The model requires significantly less speech data, as speech pretrain is skipped, which markedly simplifies the process of building a duplex S2S model from any LLMs. Finally, it is the first openly available duplex S2S model with training and inference code to foster reproducibility.

Figures

Figures reproduced from arXiv: 2505.15670 by the authors.

Figure 1
Figure 1. The proposed duplex S2S model without requiring speech-text pretraining. Our model includes a streaming speech encoder, a personalized codec, and an LLM. The model is trained to predict both text and audio channels in parallel with turn-level alignments. 3. Model Architecture To achieve duplex behavior, our S2S model takes two input streams simultaneously: user speech stream, and agent speech and text stream. As sho… view at source ↗
Figure 2
Figure 2. Duplex training data format. Our duplex data consists of separate user and agent streams including turn taking and barge-in behavior. Here, the user barges in at the second turn. 0.6 kbps with a frame rate of 12.5 frames per second, employ￾ing N = 4 independent codebooks. The codec leverages Finite Scalar Quantization (FSQ) [25], ensuring independence among codebooks. This independence removes the need for additiona… view at source ↗
Figure 3
Figure 3. Multi-turn conversation with frequent barge-in. Unseen reasoning problem. Beyond leveraging learned knowledge to generate responses, the agent also demonstrates the ability to utilize contextual information, effectively summa￾rizing the main topic of each conversation in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Spoken QA example on an unseen topic. 7. Conclusion We introduced a novel duplex S2S architecture that models si￾multaneous user and agent streams without requiring speech pretraining. Our data-efficient approach maintains end-to-end modeling of conversation reasoning …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning

    eess.AS 2026-03 unverdicted novelty 7.0 of 10

    FLAIR enables spoken dialogue AI to conduct continuous latent reasoning while perceiving speech through recursive latent embeddings and an ELBO-based finetuning objective.

  2. Unified Audio Intelligence Without Regressing on Text Intelligence

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A unified 30B MoE audio-text LLM achieves state-of-the-art audio understanding, generation, and speech tasks while preserving text reasoning comparable to its text-only backbone.

Reference graph

Works this paper leans on

45 extracted references · 15 canonical work pages · cited by 2 Pith papers

  1. [1]

    Speech, as a natural interface for human-computer interaction, is a key part of this trend

    Introduction Large language models (LLMs) [1–4] have made significant strides in natural language processing, sparking interest in mul- timodal models that extend beyond text. Speech, as a natural interface for human-computer interaction, is a key part of this trend. Recent studies suggest adapting LLMs to process speech prompts for various speech-to-text...

    work page

  2. [2]

    Key challenges here include handling simultaneous user and agent streams and enabling turn-taking

    Related Work Interest in full-duplex S2S models has grown in the past year. Key challenges here include handling simultaneous user and agent streams and enabling turn-taking. Systems like [16,18,19] model single-channel interactions but use external signals, such as stopping commands [19] or submodules [16], to decide when to respond. Models like SyncLLM ...

    arXiv 2025

  3. [3]

    As shown in Fig

    Model Architecture To achieve duplex behavior, our S2S model takes two input streams simultaneously: user speech stream, and agent speech and text stream. As shown in Fig. 1, the user speech is first encoded to generate continuous embeddings by the speech en- coder using an 80-ms frame rate. We use a 100M stream- ing speech encoder from a CTC model [21]. ...

    work page

  4. [4]

    Duplex Data for Training Table 1 summarizes our training data which can be categorized into spoken QA and multi-turn conversations. 4.1. Single-turn synthetic and real spoken QA Our most basic training data structure consists of a single-turn spoken QA between the user and agent. We use a multi-speaker TTS model [26] to synthesize the context, questions a...

    work page

  5. [5]

    Training Details We implement the model with PyTorch using the NeMo Toolkit [32], and the model is trained on 32 A100 (80G) GPUs with a batch duration of 1000 sec per GPU

    Experiment Details 5.1. Training Details We implement the model with PyTorch using the NeMo Toolkit [32], and the model is trained on 32 A100 (80G) GPUs with a batch duration of 1000 sec per GPU. The speech encoder is ini- tialized from a 100M streaming pretrained encoder with 80ms right context [21], and the LLM is initialized from the 1.1B 2A subset fro...

    work page 2024

  6. [6]

    Conversation and Speech Generation Quality We first evaluate the turn-taking and speech generation quality of our model in Table 2

    Results and Comparison 6.1. Conversation and Speech Generation Quality We first evaluate the turn-taking and speech generation quality of our model in Table 2. Compared to Moshi, our model has significantly higher barge-in success rate (94.5% v.s. 55.1%), the same false alarm rates, and lower barge-in latency (0.69s v.s. 0.81s). We observe that, in multi-...

    work page

  7. [7]

    Our data-efficient approach maintains end-to-end modeling of conversation reasoning and behaviors

    Conclusion We introduced a novel duplex S2S architecture that models si- multaneous user and agent streams without requiring speech pretraining. Our data-efficient approach maintains end-to-end modeling of conversation reasoning and behaviors. Experimen- tal results show competitive performance in reasoning, barge-in, and turn-taking. Our open-sourced tra...

    work page

  8. [8]

    Audio flamingo: A novel audio language model with few-shot learning and dialogue abili- ties,

    Z. Kong, A. Goel, R. Badlaniet al., “Audio flamingo: A novel audio language model with few-shot learning and dialogue abili- ties,”arXiv preprint arXiv:2402.01831, 2024

    arXiv 2024

Show all 45 references
  1. [9]

    Language models are few-shot learners,

    T. B. Brown, “Language models are few-shot learners,”arXiv preprint ArXiv:2005.14165, 2020

    2005 arXiv

  2. [10]

    Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wanget al., “Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context,”arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  3. [11]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwalet al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  4. [12]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Let- man, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  5. [13]

    Prompting large lan- guage models with speech recognition abilities,

    Y . Fathullah, C. Wu, E. Lakomkinet al., “Prompting large lan- guage models with speech recognition abilities,” inICASSP. IEEE, 2024, pp. 13 351–13 355

    2024

  6. [14]

    Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  7. [15]

    Salm: Speech- augmented language model with in-context learning for speech recognition and translation,

    Z. Chen, H. Huang, A. Andrusenkoet al., “Salm: Speech- augmented language model with in-context learning for speech recognition and translation,” inICASSP. IEEE, 2024, pp. 13 521– 13 525

    2024

  8. [16]

    Minmo: A multimodal large language model for seamless voice interaction,

    Q. Chen, Y . Chen, Y . Chen, M. Chen, Y . Chen, C. Deng, Z. Du, R. Gao, C. Gao, Z. Gaoet al., “Minmo: A multimodal large language model for seamless voice interaction,”arXiv preprint arXiv:2501.06282, 2025

    2025 arXiv

  9. [17]

    Chain-of- thought prompting for speech translation,

    K. Hu, Z. Chen, C.-H. H. Yang, P. ˙Zelasko, O. Hrinchuk, V . Lavrukhin, J. Balam, and B. Ginsburg, “Chain-of- thought prompting for speech translation,”arXiv preprint arXiv:2409.11538, 2024

    2024 arXiv

  10. [18]

    Audiogpt: Understanding and generating speech, music, sound, and talking head,

    R. Huang, M. Li, D. Yang, J. Shi, X. Chang, Z. Ye, Y . Wu, Z. Hong, J. Huang, J. Liuet al., “Audiogpt: Understanding and generating speech, music, sound, and talking head,” inProceed- ings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 21, 2024, pp. 23 802–23 804

    2024

  11. [19]

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities,

    D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y . Zhou, and X. Qiu, “Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities,”arXiv preprint arXiv:2305.11000, 2023

    2023 arXiv

  12. [20]

    Unified speech-text pretraining for spoken dialog modeling,

    H. Kim, S. Seo, K. Jeong, O. Kwon, J. Kim, J. Lee, E. Song, M. Oh, S. Yoon, and K. M. Yoo, “Unified speech-text pretraining for spoken dialog modeling,”arXiv preprint arXiv:2402.05706, 2024

    2024 arXiv

  13. [21]

    Glm-4-voice: Towards intelligent and human-like end- to-end spoken chatbot,

    A. Zeng, Z. Du, M. Liu, K. Wang, S. Jiang, L. Zhao, Y . Dong, and J. Tang, “Glm-4-voice: Towards intelligent and human-like end- to-end spoken chatbot,”arXiv preprint arXiv:2412.02612, 2024

    2024 arXiv

  14. [22]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D ´efossez, L. Mazar´e, M. Orsini, A. Royer, P. P´erez, H. J´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,”arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  15. [23]

    Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation,

    W. Yu, S. Wang, X. Yang, X. Chen, X. Tian, J. Zhang, G. Sun, L. Lu, Y . Wang, and C. Zhang, “Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation,”arXiv preprint arXiv:2411.18138, 2024

    2024 arXiv

  16. [24]

    Language model can listen while speaking,

    Z. Ma, Y . Song, C. Du, J. Cong, Z. Chen, Y . Wang, Y . Wang, and X. Chen, “Language model can listen while speaking,”arXiv preprint arXiv:2408.02622, 2024

    2024 arXiv

  17. [25]

    Omniflatten: An end-to- end gpt model for seamless voice conversation,

    Q. Zhang, L. Cheng, C. Deng, Q. Chen, W. Wang, S. Zheng, J. Liu, H. Yu, C. Tan, Z. Duet al., “Omniflatten: An end-to- end gpt model for seamless voice conversation,”arXiv preprint arXiv:2410.17799, 2024

    2024 arXiv

  18. [26]

    Freeze-omni: A smart and low latency speech-to-speech dia- logue model with frozen llm,

    X. Wang, Y . Li, C. Fu, Y . Shen, L. Xie, K. Li, X. Sun, and L. Ma, “Freeze-omni: A smart and low latency speech-to-speech dia- logue model with frozen llm,”arXiv preprint arXiv:2411.00774, 2024

    2024 arXiv

  19. [27]

    Mini-omni2: Towards open-source gpt- 4o with vision, speech and duplex capabilities,

    Z. Xie and C. Wu, “Mini-omni2: Towards open-source gpt- 4o with vision, speech and duplex capabilities,”arXiv preprint arXiv:2410.11190, 2024

    2024 arXiv

  20. [28]

    Be- yond turn-based interfaces: Synchronous llms as full-duplex dia- logue agents,

    B. Veluri, B. N. Peloquin, B. Yu, H. Gong, and S. Gollakota, “Be- yond turn-based interfaces: Synchronous llms as full-duplex dia- logue agents,”arXiv preprint arXiv:2409.15594, 2024

    2024 arXiv

  21. [29]

    STT En FastConformer Hybrid Transducer- CTC Large Streaming 80ms,

    NVIDIA, “STT En FastConformer Hybrid Transducer- CTC Large Streaming 80ms,” 2023, version 1.20.0, Released June 22, 2023. [Online]. Avail- able: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/ models/stt en fastconformer hybrid large streaming 80ms

    2023

  22. [30]

    Tinyllama: An open- source small language model,

    P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open- source small language model,” 2024

    2024

  23. [31]

    Nanocodec: Towards high-quality ultra fast speech llm inference,

    Anonymous, “Nanocodec: Towards high-quality ultra fast speech llm inference,”Preprint (Under Review), 2025

    2025

  24. [32]

    Nemo: a toolkit for building ai applications using neural modules,

    O. Kuchaiev, J. Li, H. Nguyenet al., “Nemo: a toolkit for building ai applications using neural modules,”arXiv preprint arXiv:1909.09577, 2019

    1909 arXiv

  25. [33]

    using the generated agent speech after removing silence

  26. [34]

    Fi- nite scalar quantization: Vq-vae made simple,

    F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen, “Fi- nite scalar quantization: Vq-vae made simple,”arXiv preprint arXiv:2309.15505, 2023

    2023 arXiv

  27. [35]

    Koel-tts: Enhancing llm based speech generation with preference alignment and classifier free guidance,

    S. Hussain, P. Neekhara, X. Yang, E. Casanova, S. Ghosh, M. T. Desta, R. Fejgin, R. Valle, and J. Li, “Koel-tts: Enhancing llm based speech generation with preference alignment and classifier free guidance,”arXiv preprint arXiv:2502.05236, 2025

    2025 arXiv

  28. [36]

    A human gen- erated machine reading comprehension dataset,

    P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Ma- jumder, A. McNamara, B. Mitra, T. Nguyenet al., “A human gen- erated machine reading comprehension dataset,”arXiv preprint arXiv:1611.09268, 2018

    2018 arXiv

  29. [37]

    Stanford alpaca: An instruction- following llama model,

    R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction- following llama model,” 2023

    2023

  30. [38]

    Instruction data generation and unsuper- vised adaptation for speech language models,

    V . Noroozi, Z. Chen, S. Majumdar, S. Huang, J. Balam, and B. Ginsburg, “Instruction data generation and unsuper- vised adaptation for speech language models,”arXiv preprint arXiv:2406.12946, 2024

    2024 arXiv

  31. [39]

    Everyday conversations for llms,

    H. Face, “Everyday conversations for llms,” https://huggingface. co/datasets/HuggingFaceTB/everyday-conversations-llama3. 1-2k, 2024

    2024

  32. [40]

    Enhancing chat language models by scaling high-quality instructional conversations,

    N. Ding, Y . Chen, B. Xu, Y . Qin, Z. Zheng, S. Hu, Z. Liu, M. Sun, and B. Zhou, “Enhancing chat language models by scaling high-quality instructional conversations,”arXiv preprint arXiv:2305.14233, 2023

    2023 arXiv

  33. [41]

    Utmos: Utokyo-sarulab system for voicemos challenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “Utmos: Utokyo-sarulab system for voicemos challenge 2022,”arXiv preprint arXiv:2204.02152, 2022

    2022 arXiv

  34. [42]

    Torchaudio-squim: Reference-less speech quality and intelligibility measures in torchaudio,

    A. Kumar, K. Tan, Z. Ni, P. Manocha, X. Zhang, E. Henderson, and B. Xu, “Torchaudio-squim: Reference-less speech quality and intelligibility measures in torchaudio,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 202...

    2023

  35. [43]

    Scaling speech technology to 1,000+ languages,

    V . Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandiet al., “Scaling speech technology to 1,000+ languages,”Journal of Machine Learning Research, vol. 25, no. 97, pp. 1–52, 2024

    2024

  36. [44]

    Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,

    E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. G ¨olge, and M. A. Ponti, “Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,” inInternational Con- ference on Machine Learning. PMLR, 2022, pp. 2709–2720

    2022

  37. [45]

    Ecapa2: A hybrid neural net- work architecture and training strategy for robust speaker embed- dings,

    J. Thienpondt and K. Demuynck, “Ecapa2: A hybrid neural net- work architecture and training strategy for robust speaker embed- dings,”arXiv preprint arXiv:2401.08342, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.