{"id":"4f8d8989-37ae-4e2f-b3a5-9e36a1c7022b","arxiv_id":"2505.15670","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A speech-to-speech language model uses channel fusion of a streaming encoder and codec tokens to handle barge-in and turn-taking without speech pretraining, showing improved metrics over Moshi at 0.6 kbps.","lead":"This paper presents a duplex speech-to-speech model that listens and speaks simultaneously using a pretrained streaming encoder instead of expensive speech pretraining. The authors report better turn-taking and reasoning than Moshi at half the speech bitrate, and they release training code and a demo.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fixed 0.64 s silence in training data (Sec. 4.1/4.2) makes the reported turn-taking and barge-in gains a learned constant-delay heuristic; real pause variability may invalidate the claimed advantage over Moshi.","rationale":"The reader's weakest assumption is also the one I would stress-test: the 0.64 s synthetic silence is used both to teach the agent when to start speaking and to model barge-in cutoff in training, and the paper's own evaluation acknowledges that response latency is a direct consequence of this constant. I do not see a deeper issue that would be more decisive for the central performance claim. The other candidate concern, the 'first without speech pretraining' priority, is a literature claim that does not affect the system's internal correctness; the paper's real contribution is the architecture and open-source release. The reasoning evaluation via GPT scores on ASR transcripts is a secondary concern because the direction of any bias is unclear and the comparison against a cascaded GT+LLM provides a sanity check. For turn-taking, however, the metric values align almost exactly with the injected training constant, and no experiment varies pause duration. This makes the headline 'outperforms Moshi in turn-taking and barge-in' conditional on real pause distributions. A focused test on natural pause distributions would settle it. If the model remains robust, the concern does not land; if not, the claim should be softened to 'matches the synthetic training distribution.' The conditional verdict remains appropriate.","tokens_in":9073,"tokens_out":4962,"duration_ms":43933,"concrete_test":"Run the released inference code on a held-out set whose user-turn-final silences and barge-in onsets are drawn from a real conversational corpus (e.g., Switchboard turn-final pause distribution) instead of the fixed synthetic 0.64 s constant, and measure barge-in success, false-alarm rate, and 1st-response latency. As a second check, retrain or fine-tune on variable silence durations (e.g., 0.2-1.5 s) and compare with the fixed-0.64 s model on the same real-pause set. If performance degrades or false-alarm rate rises sharply outside the trained delay, the reported turn-taking advantage is a training-distribution artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 4.1 states that a 0.64 s silence is inserted between user and agent before the agent speaks, and Sec. 4.2 constructs barge-in examples by cutting off agent speech and keeping exactly 0.64 s of agent audio before silence. Thus the model sees essentially one pause duration and one barge-in offset during training. The paper itself admits in Sec. 6.1 that its 1st-response latency is a direct artifact of always adding 0.64 s silence after user turns. The reported barge-in success rate of 94.5% at 0.69 s latency in Table 2 is suspiciously close to this training constant, suggesting the model reproduces a fixed delay rather than learning a general turn-taking policy. Evaluation sets are generated by the same TTS pipeline, and the Impatient set only halves the same synthetic silence; it does not exercise the variability of real user pauses, overlapping speech, or barge-in timing. A false-alarm rate of 0.0% is consistent with a model that simply never speaks during an initial user-turn interval. The central claim that the model outperforms Moshi in turn-taking and barge-in is therefore not yet supported for natural conversations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SALM-Duplex, a duplex speech-to-speech model that combines a streaming speech encoder for continuous user input with a decoder-only LLM that predicts text and low-bitrate codec speech tokens in parallel. The authors train on synthetic TTS data with multi-turn and barge-in examples, using TinyLlama-1.1B as the backbone and a 0.6 kbps personalized NanoCodec for agent speech. They compare against Moshi on barge-in performance, first-response latency, UTMOS, and GPT-based reasoning scores, and report better results on most of these metrics. The paper also releases training and inference code, and claims to be the first duplex S2S model that does not require speech-text pretraining.","tokens_in":9424,"tokens_out":5015,"duration_ms":42607,"significance":"If the results hold, the architecture is a meaningful simplification: a duplex conversational agent can be built by fine-tuning an ordinary text LLM with a streaming encoder and a low-bitrate codec, without a dedicated speech-text pretraining stage. The codec personalization result, showing that a 0.6 kbps personalized codec can outperform 1.1--1.2 kbps codecs on reconstruction metrics for the target speaker, is a useful contribution. The open-source release is a concrete strength for reproducibility. However, the empirical support for the behavioral claims is currently limited by the fixed synthetic timing used in both training and evaluation, the reliance on a single baseline, and the use of automated metrics without uncertainty quantification.","major_comments":[{"comment":"The barge-in and turn-taking evaluation is confounded by a fixed training-time timing constant. Sec. 4.1 states that 'we insert a 0.64s silence between user and agent before the agent speaks', and Sec. 4.2 keeps exactly 0.64s of agent speech before a barge-in cutoff. The Impatient set is then created by halving the same synthetic silence rather than sampling from a distribution of natural pauses or interruption offsets. The reported barge-in latency of 0.69s in Table 2 is within 0.05s of this training constant, which suggests the model may have learned a fixed-delay policy rather than a general turn-taking rule. The claimed superiority over Moshi on turn-taking and barge-in is therefore not established for natural conversations. Please evaluate with varied silence durations, real human pause distributions, and barge-in onsets, and report barge-in success and false-alarm rates as a function of interruption offset.","section":"Secs. 4.1, 4.2, and Table 2"},{"comment":"The reasoning comparison uses one GPT-based score computed on ASR transcripts, one held-out shard per dataset, no error bars or significance tests, and only Moshi as a duplex baseline. The abstract's claim that the model outperforms previous duplex models in reasoning is broader than this evidence. Please report confidence intervals across shards or bootstrap samples, and ideally add at least one additional open duplex baseline; a secondary human or reference-based metric would also strengthen the claim.","section":"Sec. 6.2 and Table 3"},{"comment":"The turn-taking metrics are not symmetric across the two systems. The false-alarm definition excludes user stops within 0.1s, and Moshi's first-response latency is listed as n/a because the authors state that Moshi almost always responds before the user finishes talking. Because the two systems operate under different turn-taking policies, comparing barge-in success rates at a fixed cutoff may be misleading. Please make the metric definitions symmetric and report the distribution of interruption timings, or use a common decision threshold for both models.","section":"Secs. 5.2 and 6.1"}],"minor_comments":[{"comment":"The phrase 'speech-texet data' appears to be a typo for 'speech-text data'.","section":"Sec. 2"},{"comment":"The evaluation uses one shard per dataset; please specify the number of conversations or utterances in each shard so readers can judge the size of the evaluation.","section":"Sec. 5.2"},{"comment":"The 'GT+LLM' baseline is labeled 'optimal', but it uses ground-truth user-turn texts; the label should be qualified as 'text-oracle' to avoid implying that it is an upper bound on all possible cascaded systems.","section":"Table 3"},{"comment":"The statement that Moshi 'almost always responds before the user finishes talking' should appear with the Table 2 caption, since it explains the n/a entry for Moshi's first-response latency.","section":"Sec. 6.1 and Table 2"},{"comment":"Table 4 reports both audio-reconstruction metrics and S2S ASR-BLEU values; please clarify which evaluation set is used for each column, since the reconstruction metrics are described on 228 test samples while ASR-BLEU appears to be computed on the ASR-QA set.","section":"Sec. 6.3 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is generally well-written and the code release is a clear strength. The main risk is that the turn-taking and barge-in evaluation is calibrated to the training-time 0.64s constant, so the headline behavioral comparisons with Moshi may not transfer to real conversations. I would ask the authors to run the additional experiments described in major comment 1 before publication. I did not find evidence of circularity: the components are evaluated as claimed, and the paper acknowledges the latency artifact in Sec. 6.1."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is architectural: they show you can build a duplex speech-to-speech model by taking an ordinary text LLM, feeding it a pretrained streaming speech encoder for the user stream, and fusing that with parallel codebook token streams for the agent. No speech-text pretraining, no word-level alignment, no separate turn-taking submodule. That combination—channel fusion plus a 0.6 kbps personalized codec—is new relative to Moshi, OmniFlatten, and MiniMo, and the fact that they ship training and inference code is a real plus for the field. Credit where due: the paper is clearly written, the codec personalization ablation is sensible, and they are honest in Sec. 6.1 that the 0.64 s inserted silence explains their own first-response latency.\n\nThe soft spots are real but not fatal. The evaluation compares against Moshi only, with no error bars, and the test sets come from the same TTS pipeline used in training. The reasoning metric is an automated GPT score on ASR transcripts, which is noisy. And the stress-test concern about the 0.64 s silence is legitimate: since every training turn uses that exact pause, the model may be learning a fixed-delay heuristic rather than a general turn-taking policy. The barge-in success rate of 94.5% at 0.69 s latency is suspiciously close to the training constant, and the Impatient set only halves the same synthetic silence. That said, barge-in does require the model to stop mid-utterance when the user interrupts, which goes beyond just after a pause, so the model is learning something more than a timer. Still, the claim of better turn-taking than Moshi in natural conversation is not yet supported.\n\nThere are also minor issues: the \"first\" claims in the abstract should be softened or scoped, since concurrent work is moving fast, and the false-alarm definition (excluding stops within 0.1 s, because Moshi is proactive) makes the 0.0% number hard to interpret.\n\nBottom line: this is a solid, useful contribution to the spoken-dialogue subfield, with an architecture worth building on and open code that makes it reproducible. It deserves a serious referee, but the quantitative evidence for the turn-taking advantage needs tightening—more baselines, real or at least varied pause distributions, and error bars. I'd bring it to reading group and would probably cite it if I worked on duplex S2S.\n\nFor peer review: accept for review, but expect the authors to address the fixed-silence issue and broaden the evaluation.","headline":"A genuinelynew duplex S2S architecture without speech-text pretraining, but the turn-taking evaluation leans on a fixed 0.64 s silence and a single baseline; worth refereeing with revisions.","tokens_in":9910,"tokens_out":1206,"would_cite":true,"duration_ms":12151,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A duplex speech-to-speech model can be built without speech-text pretraining by fusing user speech embeddings with parallel agent text and codec channels, and it beats prior duplex models on reasoning, turn-taking, and barge-in.","keywords":["duplex speech-to-speech","streaming speech encoder","barge-in","turn-taking","speech codec personalization","channel fusion","multi-channel next-token prediction","spoken dialogue"],"falsifier":"Measure barge-in success and false-alarm rates in live conversations where real users pause with natural variability instead of the fixed 0.64-second training silence; if false alarms rise sharply or success drops well below the reported 94.5%, the claimed turn-taking advantage is an artifact of the synthetic silence.","tokens_in":8851,"feed_emoji":"🗣️","tokens_out":11074,"duration_ms":84528,"temperature":0.7,"pith_summary":"The paper claims that full-duplex spoken dialogue does not require a separate speech-text pretraining stage: a pretrained streaming speech encoder can bring user audio into an ordinary text LLM, while agent speech and text are predicted in parallel as separate channels. This lowers the data and compute barrier enough that a 1.1B text model becomes a duplex voice agent, and it lets the agent voice be improved independently by fine-tuning the codec on a target speaker. The paper reports that this model outperforms the previous duplex baseline on reasoning, turn-taking, and barge-in metrics, and is competitive with an ideal cascaded text pipeline. It also makes training and inference code publicly available.","feed_headline":"Skip speech pretraining with a duplex voice AI from a text LLM","feed_subtitle":"A streaming encoder plus 0.6 kbps codec turns any text LLM into a full-duplex talker that handles barge-in.","key_machinery":"The central object is channel fusion of two simultaneously active streams in a decoder-only LLM: user embeddings from a pretrained 100M streaming encoder at 80 ms frames are summed with agent token embeddings made of text tokens and four parallel 12.5 Hz codec codebooks. A one-token delay on the speech channels lets text lead slightly. The codec's four independent codebooks, built with Finite Scalar Quantization, are predicted in parallel at each timestep, and the whole model is trained by multi-channel next-token prediction with text loss weighted 3 and speech loss weighted 1. Turn-level alignment of text and speech with separate start and end tokens and silence padding is what turns ordinary QA and conversation data into a duplex training format.","core_discovery":"The central claim is that duplex conversation can be modeled directly: continuous user speech is encoded by a pretrained streaming encoder, and those embeddings are summed with parallel agent text and codec-speech token embeddings inside a causal LLM, so the model predicts what the agent says and how it sounds at the same time. Agent speech is tokenized with a 0.6 kbps codec that uses four independent parallel codebooks, and text and speech channels are aligned at the turn level with their own start and end tokens. Because the agent stream and user stream are handled by separate modules, the codec can be fine-tuned on a target voice without retraining the dialogue model. The paper's results show higher barge-in success, lower barge-in latency, equal false-alarm rate, and better reasoning scores than the previous duplex baseline, while using a smaller backbone and no speech-text pretraining.","pith_inferences":["If the central claim holds, the practical cost of building a duplex voice assistant shifts almost entirely to data curation and codec voice selection, because the model architecture reduces to fine-tuning an existing text LLM.","One testable extension is to draw turn-taking silence durations from real conversation corpora rather than the fixed 0.64-second value, which could improve transfer to users with atypical pause patterns.","Because user and agent are modeled as separate summed channels, the same fusion mechanism could be extended to multi-speaker or multi-agent dialogue by adding more channels, not by changing the learning objective.","A direct ablation would replace the pretrained streaming encoder with a randomly initialized one; if performance collapses, the 'no speech pretraining' claim is really carried by the encoder's pretrained speech understanding."],"forward_implications":["Any text LLM can be turned into a duplex voice agent by fine-tuning with a streaming encoder and codec speech tokens, since no speech-text pretraining step is required.","Agent voice quality is decoupled from dialogue reasoning: fine-tuning the codec on a target speaker improves both audio reconstruction quality and the word-overlap score computed from transcripts of generated speech, even at half the bitrate of prior codecs.","The barge-in behavior transfers from synthetic silence-based data to evaluation: on an impatient-user set the model stops speaking within 1.5 seconds in 94.5% of interruptions, compared with 55.1% for the prior duplex baseline.","Reasoning quality is competitive with an optimal cascaded system that feeds ground-truth user text into an LLM, while avoiding the latency and pipeline complexity of cascades.","Open training and inference code means the recipe can be reproduced and adapted by others without rebuilding the data pipeline from scratch."],"supporting_citations":[{"why":"The prior duplex speech-text foundation model used as the main comparison for reasoning, turn-taking, and barge-in.","marker":"[14]"},{"why":"The 100M streaming speech encoder pretrained from a CTC model supplies user input embeddings and removes the need for speech pretraining.","marker":"[21]"},{"why":"The 1.1B text language model used as the backbone LLM for initialization and fine-tuning.","marker":"[22]"},{"why":"The low-bitrate codec supplying four parallel speech codebooks, used also for agent-voice personalization.","marker":"[23]"},{"why":"The source of the channel-fusion idea of summing user and agent embeddings in a language model that listens while speaking.","marker":"[24]"},{"why":"The quantization method that keeps codebooks independent, enabling parallel codebook prediction and low latency.","marker":"[25]"},{"why":"The multi-speaker TTS model used to synthesize duplex training data from text conversations.","marker":"[26]"},{"why":"The multi-turn chat dataset supplying conversational content for synthetic dialogue training and evaluation.","marker":"[31]"}],"fun_headline_variants":["Direct duplex S2S: skip speech pretraining, handle barge-in","First open duplex S2S model without speech pretraining","Streaming encoder plus 0.6 kbps codec for duplex speech","From text LLM to duplex voice: no speech pretraining needed","Duplex speech modeling: fuse user and agent streams directly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The learned turn-taking and barge-in behavior depends on a fixed 0.64-second silence inserted between user and agent turns in all training data, so if real user pause patterns differ from this synthetic constant, the reported barge-in and false-alarm trade-off may not transfer to real conversations.","fun_headline_variants_meta":{"raw":{"variants":["Direct duplex S2S: skip speech pretraining, handle barge-in","First open duplex S2S model without speech pretraining","Streaming encoder plus 0.6 kbps codec for duplex speech","From text LLM to duplex voice: no speech pretraining needed","Duplex speech modeling: fuse user and agent streams directly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2837,"prompt_tokens":918,"completion_tokens":1919,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":1828}},"tokens_in":534,"tokens_out":1919,"duration_ms":12936,"temperature":1.0,"reasoning_tokens":1828,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:12:00.690084+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure barge-in success and false-alarm rates in live conversations where real users pause with natural variability instead of the fixed 0.64-second training silence; if false alarms rise sharply or success drops well below the reported 94.5%, the claimed turn-taking advantage is an artifact of the synthetic silence.","supporting_citations":[{"cited_title":"Nanocodec: Towards high-quality ultra fast speech llm inference,","cited_arxiv_id":null,"evidence_quote":"The multi-turn chat dataset supplying conversational content for synthetic dialogue training and evaluation."}],"review_version":1}