{"id":"7f9a954d-de5e-4ef8-8f4f-8b77f209c16f","arxiv_id":"2411.13577","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"WavChat categorizes spoken dialogue models into cascaded and end-to-end paradigms and surveys speech representations, training strategies, streaming, duplex interaction, datasets, and evaluation benchmarks.","lead":"This survey maps spoken dialogue models, the systems that let AI take voice input and reply in speech, into cascaded and end-to-end families and reviews their building blocks, training data, and benchmarks. It is a reference for tracking how voice AI is moving from separate speech-to-text, text-model, and text-to-speech modules toward single speech-native models.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cascaded/end-to-end taxonomy is applied inconsistently: Freeze-Omni and Llama-Omni are classed as end-to-end although their LLMs do not directly emit speech tokens, so the paper's central categorization is ambiguous.","rationale":"The reader's conditional verdict rests mainly on unresolved coverage and selection issues: the absence of a documented inclusion protocol and the explicit exclusion of named unpublished systems from Figure 1. That is a legitimate concern about the survey's completeness. My stress-test identifies a different, and arguably more central, weakness: the paper's own definition of the end-to-end category is not applied consistently to the systems it includes. The survey's contribution is not a new model or measurement but a taxonomy and a field map. If the boundary between cascaded and end-to-end systems cannot be applied mechanically from the stated definition, then the map is less reliable even if coverage were complete. I do not think this invalidates the survey: the taxonomy can be repaired by stating the intended criterion precisely, splitting the current 'end-to-end' category into subcategories such as 'LLM-token-output' vs. 'separate speech decoder', and reclassifying the affected systems. The reader's verdict of CONDITIONAL therefore remains appropriate, and my concern reinforces that decision rather than changing it. I credit the paper for its broad citation base, its explicit admission of the unpublished-system exclusions, and its structured treatment of representations, training, streaming, and evaluation; those strengths make the inconsistency fixable in revision. My agreement is 'partial' because the reader focused on external coverage while I focus on internal definitional consistency: both concern the reliability of the map, but they are distinct failure modes.","tokens_in":42778,"tokens_out":6296,"duration_ms":72798,"concrete_test":"Reclassify every system in Figure 1 and Figure 2 using the operational test stated in Section 2.3: does the LLM's own output layer emit speech tokens as part of its autoregressive vocabulary, or does a separate decoder produce speech tokens from LLM hidden states? Apply this rule to Freeze-Omni, Llama-Omni, IntrinsicVoice, and SpeechGPT. If Freeze-Omni and Llama-Omni fail the LLM-output-layer test but are still labeled end-to-end, then the paper must either revise the definition to a two-axis taxonomy (e.g., text-intermediacy vs. output-generator location) or reclassify these systems; a per-system table with the criterion used for each entry would settle the ambiguity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The survey's central contribution is a binary categorization: cascaded if the core LLM cannot directly understand and generate speech representations, and end-to-end if it can. This criterion is stated in Section 1 and again in Section 2.3. The paper then applies it inconsistently. In Section 2.3, Freeze-Omni is presented as an end-to-end system whose 'core idea lies in transferring the functionalities of spoken dialogue models to the encoder (ASR) and decoder (TTS), rather than assigning these tasks to the large language model'; Llama-Omni is also listed as end-to-end, yet its streaming speech decoder takes LLM hidden states as input and generates discrete speech units outside the LLM output layer. Under the paper's own definition, these systems do not have their core LLM directly generating speech representations. The same blur appears with SpeechGPT: it autoregressively generates text tokens before speech tokens, and Section 3.3.4 concedes that such text-guided generation uses intermediate text as a semantic guide, yet it is classified as end-to-end rather than as a hybrid. The result is that 'end-to-end' sometimes means 'no external ASR/TTS modules', sometimes 'no text intermediacy', and sometimes 'the LLM emits speech tokens as its own output'. These are different properties, and the paper does not separate them. Because the value of the survey is precisely a reproducible map of the field, this ambiguity is a load-bearing correctness risk: a reader cannot independently apply the stated rule to reproduce the figure's assignments. This is an internal inconsistency, not merely a disagreement with an external taxonomy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript surveys spoken dialogue models (SDMs) as of late 2024. It proposes a binary taxonomy that separates cascaded from end-to-end systems based on whether the core language model can directly understand and generate speech representations, presents a chronological timeline of systems in Figure 1, and then reviews speech representations, training paradigms, streaming/duplex/interaction mechanisms, and training/evaluation resources. It does not propose a new model or report new experimental results; its contribution is the organization, comparison, and discussion of existing systems.","tokens_in":43063,"tokens_out":7639,"duration_ms":77037,"significance":"If its taxonomy were applied consistently and its coverage criteria documented, this would be a useful reference for a rapidly growing area. The paper assembles a broad catalog of systems, organizes architectural choices around representations, alignment, training stages, duplex/streaming behavior, and benchmarks, and ends each section with a discussion of open problems. The explicit separation of semantic and acoustic representations and the summary of existing evaluation resources are genuinely useful. No machine-checked proofs or reproducible code are involved because the contribution is a survey; the associated GitHub repository is a helpful pointer but is not itself a reproducibility artifact. The main threat to the paper's value is the inconsistent application of its central categorization, which currently prevents a reader from using the map as the authors intend.","major_comments":[{"comment":"The paper's central definition of end-to-end is stated as 'whether the core language model can directly understand and generate speech representations' (§1) and repeated in §2.3, yet several systems placed in the end-to-end column do not satisfy this criterion. §2.3 describes Freeze-Omni as transferring 'the functionalities of spoken dialogue models to the encoder (ASR) and decoder (TTS), rather than assigning these tasks to the large language model,' and describes Llama-Omni and IntrinsicVoice as feeding 'the hidden states of the LLM' into a separate decoder. Hidden states are not speech representations, and a TTS decoder is not the LLM generating speech. The same ambiguity appears for SpeechGPT, which generates text before speech (§2.3) and is later described in §3.3.4 as using intermediate text as a semantic guide. The manuscript therefore uses 'end-to-end' to mean at least three different properties: no external ASR/TTS modules, no text intermediacy, and LLM-emitting speech tokens. Please either narrow the definition, introduce a third (hybrid/hidden-state) category, or explicitly classify these systems along all three axes; otherwise the central map cannot be reproduced by a reader.","section":"§1, §2.3, Figure 2"},{"comment":"The abstract claims that the paper is the first to compile existing spoken dialogue systems in chronological order, but the paper never states a search or inclusion/exclusion protocol. Figure 1's caption says systems without published papers (Westlake-Omni, MooER-Omni, Hertz-dev, SpeechGPT2, Fish-Agent) are excluded even though some are named in the text and are part of the field; the timeline is therefore not a complete map of 'existing' systems. To make the survey's coverage claim falsifiable, please add a scope-and-methods paragraph stating the date of the literature search, the sources searched, the criteria for including a system (e.g., public demo, technical report, checkpoint availability), and how the chronological order is determined (arXiv submission date vs release date). As written, the selection appears to be subjective, and a reader cannot tell whether the omissions are material.","section":"§1, Figure 1 caption"},{"comment":"The paper's own subcategorizations conflict with its headline taxonomy. §4.1 places Llama-Omni under 'Parallel Generation of Text and Speech' rather than 'Speech-to-Speech Generation,' while §2.3 places it in the end-to-end group; §5.1.1 states that several systems named in the end-to-end group 'cannot be considered fully streaming because they are not designed to be streaming on the input side.' These statements are not errors in themselves, but they show that the binary 'cascaded vs end-to-end' distinction is orthogonal to at least two other distinctions (text-guided vs text-free; streaming input vs streaming output) that the paper does not separate. Please add a table or figure that crosses the taxonomy with these additional axes so that systems like Llama-Omni, Mini-Omni, and IntrinsicVoice are classified unambiguously.","section":"§2.3, §4.1, §5.1.1"}],"minor_comments":[{"comment":"Citation numbers for Freeze-Omni are inconsistent: it is cited as [213] in §2.3 and as [214] in §3.1, §3.2, and §5.2.2. Please unify the reference numbers and verify that the bibliography entries correspond.","section":"§2.3, §3.1, §3.2, §5.2.2"},{"comment":"Several typos should be corrected: 'noly' should be 'only' (§2.3), 'interactation' should be 'interaction' (§5), 'moded' should be 'model' (§5.2.1), and 'futher' should be 'further' (§3.2).","section":"Throughout"},{"comment":"System names are not standardized: 'SALMONN' appears as 'SALMMON' (§4.1) and 'SLAMONN' (Figure 3), and 'Llama-Omni' appears as 'LLama-Omni' and 'Lama-Omni' (§4.3). Please standardize names to the official spellings.","section":"Figure 3, §4.1, §4.3"},{"comment":"The caption says that publicly available checkpoints are marked in yellow, but the figure does not contain a legend, and the text does not define the marking scheme. Please add a legend and clarify that the timeline includes only systems with published papers, so the reader is not misled about coverage.","section":"Figure 1"},{"comment":"The row for Wenetspeech is labeled 'TTS' although WenetSpeech is widely used as an ASR corpus; please correct the task label and re-check all task assignments in the table (e.g., VCTK, Common Voice, and Gigaspeech) for accuracy.","section":"Table 2"},{"comment":"The manuscript is marked 'Working in Progress.' This label is not appropriate for a journal submission; please remove it and bring the manuscript to a finished state, including the missing search protocol described in the major comments, before resubmission.","section":"Title page"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a self-described working-progress arXiv preprint. The central taxonomy, which is the paper's main contribution, needs to be made internally consistent, and the coverage claims need a documented selection protocol. I would not support rejection if the authors are willing to make these revisions. One additional concern for the editor: the paper gives detailed individual treatment in §4.3 to several systems authored by members of the same group (Mini-Omni, Mini-Omni2, OmniFlatten), while comparable third-party systems are often only listed; this does not by itself invalidate the taxonomy, but a neutral coverage check would strengthen the survey's credibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe first thing you should know about WavChat is that it is genuinely useful: it is the first survey I know that puts the recent explosion of spoken dialogue models into one chronological map, and the four-part organization (representations, training, streaming/duplex, data and evaluation) is a sensible frame. The dataset and benchmark tables alone are worth the download. But the central taxonomy, the cascaded/end-to-end split, is not applied as cleanly as the paper's own definition promises. The definition in Section 2.3 says end-to-end means the core LLM directly understands and generates speech representations. Then Freeze-Omni is called end-to-end even though the paper says its core idea is to move the functions to the ASR encoder and TTS decoder rather than the LLM, and Llama-Omni is listed as end-to-end while its speech decoder generates units from LLM hidden states outside the LLM's output layer. SpeechGPT is also end-to-end by the paper's classification, even though it generates text before speech tokens. So 'end-to-end' is doing different work in different places. This matters because the whole value of a survey like this is that a reader can use the map without having to rederive it. As written, I couldn't reproduce the figure's assignments from the stated rule. That is a real weakness, and I agree with the stress-test that it is the biggest single risk to the paper's usefulness.\n\nThere are also smaller issues: no documented selection protocol, Figure 1 omits several named systems (Westlake-Omni, MooER-Omni, etc.), and there are citation inconsistencies (Freeze-Omni is [213] in one place and [214] in another). The paper is marked 'Working in Progress', which explains some of this, but a survey that claims comprehensiveness needs to say how it chose its systems.\n\nWhat the paper does well is substantial. The discussions of semantic vs acoustic representations, the multi-stage training breakdown, and the streaming/duplex interaction analysis are accurate and well-sourced. The chronological timeline up to late 2024 is useful. I did not find evidence of misrepresentation of the underlying systems, aside from the classification issue above.\n\nWho is this for? Someone entering the spoken dialogue area, or a researcher wanting a quick reference for what systems exist and how they are built. It won't change the direction of the field, but it can save a newcomer weeks of digging.\n\nMy recommendation: send it to peer review. The taxonomy needs a serious revision (perhaps a third 'hybrid' category or an explicit separation of text-guided vs text-free end-to-end), the selection criteria need to be documented, and the citations need a cleanup. But the underlying work is solid and the field needs this kind of map. If the authors fix the categorization, I would cite it without hesitation.\n\nThat's my read.","headline":"Useful survey with a taxonomy that doesn't always apply; worth reviewing after fixing the categorization and selection transparency.","tokens_in":43669,"tokens_out":5184,"would_cite":true,"duration_ms":48865,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey organizes spoken dialogue systems into cascaded and end-to-end paradigms and maps the technologies behind them.","keywords":["spoken dialogue models","cascaded speech systems","end-to-end speech models","speech representations","full-duplex interaction","streaming speech","training paradigms","speech evaluation benchmarks"],"falsifier":"A reader could run a systematic search for spoken-dialogue systems announced in the same period and count how many influential ones are missing from the paper's timeline, especially releases without published papers that the text itself names; if the omissions change the narrative of how the field progressed, the claim to a complete chronological compilation is weakened.","tokens_in":42566,"feed_emoji":"🎙️","tokens_out":6100,"duration_ms":59051,"temperature":0.7,"pith_summary":"This survey sets out to give the field of spoken dialogue models its first systematic map. It compiles the systems released in recent years in chronological order and splits them into two paradigms: cascaded models, which route speech through text using ASR, an LLM, and TTS, and end-to-end models, whose core language model directly understands and generates speech representations. Around that split, it organizes the underlying technology into four areas—speech representation, training paradigm, streaming/duplex/interaction, and data/evaluation—and closes each with open problems. The value of the map, if it holds, is that researchers and engineers can locate any system on a common grid, compare design choices, and see where the gaps are. The paper does not claim a new model or a measured result; its contribution is the organization and vocabulary.","feed_headline":"Survey maps spoken dialogue models into two paradigms","feed_subtitle":"WavChat sorts cascaded and end-to-end systems, then reviews the speech tokens, training, streaming, and benchmarks behind them.","key_machinery":"The load-bearing device is the cascaded/end-to-end dichotomy, defined by whether the core LLM can directly understand and generate speech representations rather than relying on text as an intermediate. The survey's secondary machinery is a four-part decomposition of the field: speech representations split into semantic tokens (HuBERT, Whisper, S3) and acoustic tokens (EnCodec, SpeechTokenizer, Mimi); training paradigms arranged as five architecture types plus a four-stage pipeline (text LLM pretraining, modality alignment, supervised fine-tuning, preference optimization); streaming and interaction techniques such as causal convolution, causal attention, queue management, and simplex/half-/full-duplex modes; and a catalogue of datasets, evaluation metrics, and benchmarks. This grid carries the argument by giving every surveyed system a location and making the field's open problems visible.","core_discovery":"The paper's central claim is that the rapid recent growth in spoken dialogue systems can be understood through one organizing question: does the core language model directly understand and generate speech representations? Systems that answer yes are classified end-to-end; systems that only generate text and convert it to speech are classified cascaded. Using this criterion, WavChat assembles the systems chronologically and then reviews four interlocking technology areas (speech representations, training paradigms, streaming/duplex/interaction design, and training/evaluation resources), describing limitations and future directions in each. The survey presents itself as the first to organize the field this way and as a reference point for both academic research and industrial applications.","pith_inferences":["The binary cascaded/end-to-end split may become less crisp as cascaded systems gain trainable streaming TTS and end-to-end systems keep implicit text guidance; a graded degree-of-text-mediation axis might be a more faithful map.","The exclusion from the timeline of systems without published papers (e.g., Westlake-Omni, MooER-Omni, Hertz-dev, SpeechGPT2, Fish-Agent) means the map can lag the deployed frontier; tracking paper-less releases as they appear would test the timeline's completeness.","The survey's representation discussion suggests a testable prediction: codecs that distill semantic information into the first quantizer layer (as SpeechTokenizer and Mimi do) will become the default speech tokenizer for end-to-end dialogue, because they resolve the semantic-versus-acoustic tradeoff before the LLM sees the tokens.","The absence of RL and preference optimization in spoken dialogue, contrasted with its role in TTS, points to an extension: preference data defined on interaction outcomes (interrupt handling, backchannels, latency) rather than speech quality alone."],"forward_implications":["New systems can be classified quickly by whether the core LLM directly consumes or produces speech tokens, making comparisons across papers more systematic.","The four-part grid functions as a design checklist: representation choice, alignment and training strategy, streaming and duplex handling, and evaluation plan.","The survey identifies concrete open gaps: no unified framework for all interaction forms, scarce RL and preference optimization in spoken dialogue, few models beyond English and Chinese, and underdeveloped audio/music generation and security evaluation.","The chronological timeline frames the field's trajectory as moving from text-intermediary pipelines (AudioGPT) through speech-token LLMs (SpeechGPT) to full-duplex streaming systems (Moshi, Mini-Omni2).","Because most systems are built on pretrained text LLMs with speech-text alignment, the survey implies that preserving text intelligence while adding speech is the central training problem."],"supporting_citations":[{"why":"Supplies the earliest cascaded prototype (ASR plus LLM plus TTS) that anchors the text-intermediary paradigm.","marker":"[85]"},{"why":"Provides the first fully end-to-end, text-free spoken dialogue system, anchoring the end-to-end paradigm and duplex interaction.","marker":"[158]"},{"why":"Supplies the modern reference end-to-end full-duplex system, including the Mimi codec, RQ-Transformer, Inner Monologue, and latency measurements.","marker":"[44]"},{"why":"Establishes autoregressive generation of text plus speech tokens, the chain-of-modality approach many later systems build on.","marker":"[243]"},{"why":"Represents the cascaded multitask speech-text LLM line used to illustrate speech and audio understanding and preference optimization.","marker":"[33]"},{"why":"Provides delayed parallel decoding for simultaneous text and speech generation, a core streaming and duplex technique.","marker":"[223]"},{"why":"Contributes the flattening and multi-stream training approach and block-based streaming used for full-duplex, text-free dialogue.","marker":"[247]"},{"why":"Supplies one of the main benchmarks (VoiceBench) that the evaluation section organizes around.","marker":"[29]"}],"fun_headline_variants":["Spoken dialogue models: cascaded vs end-to-end","Survey splits spoken dialogue models into two camps","Two paradigms guide the spoken dialogue survey","WavChat: mapping the two ways to build speech AI","Cascaded or end-to-end? A survey of spoken dialogue"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey assumes that the systems it chose to include, selected without a documented search or inclusion protocol, are representative enough of the whole field to support a comprehensive chronological map and technology review.","fun_headline_variants_meta":{"raw":{"variants":["Spoken dialogue models: cascaded vs end-to-end","Survey splits spoken dialogue models into two camps","Two paradigms guide the spoken dialogue survey","WavChat: mapping the two ways to build speech AI","Cascaded or end-to-end? A survey of spoken dialogue"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001152,"raw_usage":{"total_tokens":4775,"prompt_tokens":942,"completion_tokens":3833,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":3756}},"tokens_in":558,"tokens_out":3833,"duration_ms":30998,"temperature":1.0,"reasoning_tokens":3756,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:08:48.570667+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could run a systematic search for spoken-dialogue systems announced in the same period and count how many influential ones are missing from the paper's timeline, especially releases without published papers that the text itself names; if the omissions change the narrative of how the field progressed, the claim to a complete chronological compilation is weakened.","supporting_citations":[],"review_version":1}