{"id":"7a402f2a-6ac7-4f23-aa2c-d1505855e583","arxiv_id":"2501.00805","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A hybrid pipeline (LLM to text, duration predictor, phoneme-conditioned speech language model) generates spoken dialogue with improved semantic coherence while preserving turn-taking naturalism.","lead":"SLIDE combines a large language model with a speech language model to generate spoken dialogues that are both semantically coherent and naturalistic. It offers a practical path for creating spontaneous-sounding conversational speech with controllable content.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic-coherence claim is confounded: SLIDE conditions on phoneme sequences derived from the LLM text, so the reported perplexity largely measures GPT-4o text quality, not the integrated speech model; the missing cascaded baseline invalidates the 65.8% reduction claim.","rationale":"I focus on the semantic-coherence evaluation because it is the paper's headline quantitative result and the basis for the central claim. The phoneme-conditioning naturalism concern raised by the reader is real, but the evidence there is mixed: N-MOS is comparable to dGSLM and objective turn-taking statistics are within ranges, though the overlap-capping post-processing is a warning sign. The semantic metric, by contrast, is systematically confounded in a way that makes the main claim untestable from the reported numbers. Since the generated audio is conditioned on phoneme tokens derived from the very text whose coherence is measured, the PPL measures the LLM's text distribution, not the spoken-dialogue model's semantic ability. The missing cascaded baseline is the decisive gap; without it, one cannot attribute the improvement to the integration rather than to GPT-4o. I therefore agree with the reader's conditional verdict but differ on which assumption is weakest: the evaluation confound is more load-bearing than the phoneme-duration naturalism assumption. If the concrete test shows that a plain TTS of the same GPT-4o text yields similar PPL, then the paper's first contribution ('incorporating LLMs') is doing all the semantic work, and the claim should be reframed. If, instead, the cascaded TTS PPL is much worse (e.g., because Whisper errors or TTS rhythm hurt), then the SLIDE pipeline would have some genuine semantic contribution. This is an empirical question that the current paper does not answer.","tokens_in":7987,"tokens_out":6104,"duration_ms":54854,"concrete_test":"Compute the same first-50-word DialoGPT perplexity on (1) the raw GPT-4o text continuations before any speech synthesis, and (2) audio produced by a standard cascaded system (e.g., a state-of-the-art neural TTS reading the same GPT-4o text) after identical Whisper transcription and tokenization. If either PPL is statistically indistinguishable from SLIDE-2's 421.29, the semantic-coherence advantage over dGSLM is inherited from the LLM and not attributable to the SLIDE speech-generation pipeline. A subsidiary check: report the ASR word error rate between the generated audio and the conditioning text; a low WER confirms the lexical content is dictated by the text, making the PPL a text-quality measure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SLIDE 'significantly improves semantic coherence' over dGSLM rests on DialoGPT perplexity of Whisper transcripts of generated audio (Section III-B, Table III). However, in SLIDE the audio is generated by conditioning dGSLM on spoken phoneme sequences derived directly from the GPT-4o text (Sections II-A, II-B, II-C). This fixes the lexical content before acoustic generation; the transcript's first-50-word perplexity therefore mostly reflects the predictability of the LLM-written text under DialoGPT, not any capability of the proposed duration/dGSLM module. The comparison to dGSLM (1228.82) is not head-to-head: dGSLM must generate both content and prosody from speech units, while SLIDE is given the content through conditioning. No cascaded LLM+TTS baseline is reported ('-' in Table III), and no significance testing is provided. The paper itself states the conditioning 'constrains the output speech units ... ensuring semantic coherence' (Section II), which acknowledges that coherence is inherited rather than learned. Consequently, the headline PPL reduction and the 11.9% gap to ground truth do not establish that the integrated model—as opposed to the LLM—improves semantic coherence. The naturalism claim is also weakened by the post-hoc silence insertion that caps excessive overlaps (Section II-B), but the primary load-bearing issue is the confounded semantic evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SLIDE, a hybrid spoken-dialogue generation system that combines a text-based LLM with a textless speech language model. GPT-4o generates textual dialogue continuations from a transcribed prompt; the text is converted to phoneme sequences via G2P; a two-tower transformer duration predictor, trained on forced-alignment durations, converts written phoneme sequences into spoken phoneme sequences; and a dGSLM-style model conditioned on those spoken phoneme sequences generates audio tokens that are decoded by HiFi-GAN. Experiments on Fisher compare SLIDE-1 (ground-truth text conditioning), SLIDE-2 (LLM text conditioning), dGSLM, a cascaded system, and ground truth, using turn-taking event statistics, DialoGPT perplexity of the first 50 words of transcripts, and MOS ratings for naturalness and meaningfulness. The paper's headline claims are a 65.8% relative perplexity reduction over dGSLM (1228.82 to 421.29), a small 11.9% gap to ground-truth perplexity, and naturalness comparable to dGSLM while substantially improving meaningfulness.","tokens_in":8256,"tokens_out":7748,"duration_ms":72273,"significance":"If the claims were fully supported, SLIDE would be a practical and interesting recipe for injecting LLM-level semantic control into textless speech language models while retaining their turn-taking and paralinguistic behaviors. The paper builds on established components (dGSLM, HuBERT, HiFi-GAN, G2P, Montreal Forced Aligner), evaluates on a standard large corpus, and reports subjective MOS with multiple raters, which are strengths. The central limitation is that the semantic-coherence evaluation does not isolate the contribution of the proposed speech-side components from the semantic content supplied by the LLM text, and the missing cascaded baseline prevents the reader from judging whether the improvement over dGSLM is due to the integrated system or simply to using GPT-4o for content.","major_comments":[{"comment":"The headline semantic-coherence result is confounded. In SLIDE, the audio is generated by conditioning dGSLM on spoken phoneme sequences derived from the LLM text (described in Sections II-A through II-C), so the DialoGPT perplexity of Whisper transcripts of the generated audio is largely determined by the predictability of the GPT-4o text, not by the proposed duration predictor or conditioned dGSLM. Because Table III reports '-' for the Cascaded baseline, the 65.8% relative perplexity reduction from 1228.82 to 421.29 does not establish that the integrated system improves coherence beyond a conventional LLM+TTS cascade. Please add the cascaded perplexity, report the perplexity of the LLM text before synthesis, and include an ablation in which the conditioning phoneme sequence is mismatched or the LLM text is replaced by a lower-coherence baseline.","section":"Section III-B, Table III"},{"comment":"The perplexity numbers are presented as point estimates with no sample size, variance, or significance test, yet the text uses the word 'significantly' for the 65.8% reduction and for the 11.9% gap to ground truth. Report the number of generated dialogues, the per-dialogue perplexity distribution or bootstrap confidence intervals, and a paired significance test against dGSLM and ground truth. Also justify the choice to truncate transcripts to the first 50 words, since that window may not reflect the coherence of the full dialogue.","section":"Section III-B, Table III"},{"comment":"The naturalism claim is weakened by the post-processing step that inserts silence tokens to cap overlaps, and the asserted similarity to ground truth is not quantitatively tested. SLIDE-2 has 15.8 overlaps per minute and 7.0 seconds of overlap per minute versus 8.2 and 4.0 seconds for ground truth, roughly double the ground-truth rate. Please report how often the post-processing triggers, present results without it, and provide significance tests or effect sizes for the turn-taking statistics rather than relying only on visual inspection of box plots.","section":"Section II-B, Tables I and II"},{"comment":"The subjective M-MOS improvement over dGSLM (4.08 versus 1.52, described as a 270.0% relative improvement) is reported without a significance test, and the same content-confounding applies because raters hear audio generated from LLM-conditioned text. Please report the number of rated samples, the number of raters per sample, inter-rater agreement, and a paired significance test. This is load-bearing for the claim that the proposed method 'significantly enhances the meaningfulness' of generated dialogues.","section":"Section III-C3, Table III"}],"minor_comments":[{"comment":"The heading 'Written Phoneme Sequence Duartion Prediction' contains a typo; it should read 'Duration'.","section":"Section II-B heading"},{"comment":"The heading 'Implemention Details' contains a typo; it should read 'Implementation'.","section":"Section III-A heading"},{"comment":"The rule 'If the phoneme generated in the current time step differs from the penultimate generated phoneme' is ambiguous: it is unclear whether the comparison should be against the immediately previous phoneme or the one before it, and this detail matters for reproducibility.","section":"Section II-B, inference paragraph"},{"comment":"There is a spacing typo in the Ground Truth M-MOS entry ('4 .63' instead of '4.63').","section":"Table III"},{"comment":"The sentence 'The generation temperature is always set to 1' is ambiguous because it does not specify which generation stage (LLM, duration predictor, dGSLM, or DialoGPT evaluation) the temperature applies to; please clarify.","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"This is a systems-oriented paper that could be suitable for a speech conference or journal after the evaluation is strengthened. The main risk is that the semantic-coherence claim is currently supported only by a comparison that conflates the LLM's text-generation ability with the proposed speech-side model. The missing cascaded baseline and the absence of statistical testing are the key blockers. There is also a reproducibility concern with the use of a proprietary, version-unspecified GPT-4o API, although the provided demo page mitigates this somewhat."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SLIDE is a real engineering effort: the authors take dGSLM and condition it on phoneme sequences derived from LLM-generated text, with a learned duration predictor so the phasing of the two channels lines up. That is a sensible way to inject content into a textless SLM, and the naturalism results suggest it mostly works: turn-taking stats and N-MOS are close to both dGSLM and ground truth. Credit where due — the two-tower duration predictor is a reasonable solution to a genuine alignment problem, and the M-MOS jump from 1.52 to 4.08 is hard to dismiss.\n\nThe soft spot is the semantic-coherence evaluation, and it is load-bearing. The perplexity comparison (1228.82 vs 421.29) looks dramatic, but SLIDE is given the text through phoneme conditioning, so DialoGPT is essentially scoring GPT-4o's transcript. dGSLM has to invent content from speech units; SLIDE is handed it. That is not a head-to-head comparison. The 'Cascaded' row in Table III is blank, which is the exact baseline you need: LLM text plus a standard TTS. Without it, the 65.8% reduction and the 11.9% gap to ground truth are not evidence about the integrated model. Also, perplexity is computed on the first 50 words only, with no significance testing, and the MOS intervals have only five raters per sample. These are fixable, but they need fixing.\n\nThe overlap post-processing (inserting silences when overlap exceeds 0.6s) is a minor heuristic, fine as an implementation detail but worth flagging as a tunable part of the pipeline.\n\nWho is this for? People building spoken dialogue systems who want a concrete recipe for combining LLM semantics with textless SLM naturalism. It is a useful system paper, not a methodological breakthrough. My take: send it to review, but the referee should require a cascaded baseline and proper statistics before the semantic claims are accepted.","headline":"SLIDE is a useful hybrid system, but its headline semantic-coherence numbers mostly measure the LLM's text quality, not the integrated model.","tokens_in":8816,"tokens_out":3082,"would_cite":true,"duration_ms":27911,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Conditioning a textless speech language model on duration-annotated phoneme sequences generated by an LLM produces spoken dialogue that is nearly as coherent as real conversation while staying as natural as the original textless model.","keywords":["spoken dialogue generation","speech language models","textless SLM","semantic coherence","naturalism","duration prediction","LLM-conditioned speech","discrete speech units"],"falsifier":"Take a held-out set of Fisher dialogues, run the SLIDE-2 pipeline end to end, transcribe the generated audio with Whisper, and compute DialoGPT perplexity against dGSLM output: the central claim collapses if the 65.8% relative reduction is not reproduced, or if a listener panel rates SLIDE-2 naturalness below dGSLM's 4.14 by a margin exceeding the confidence interval.","tokens_in":7782,"feed_emoji":"🗣️","tokens_out":7354,"duration_ms":65464,"temperature":0.7,"pith_summary":"SLIDE is a hybrid pipeline for generating spontaneous spoken dialogue: an LLM writes the dialogue text, a two-tower transformer predicts how long each phoneme lasts, and the speech language model dGSLM is conditioned on the resulting “spoken phoneme sequence” (each phoneme repeated every 20 ms) to vocalize it. The paper's central claim is that this text-side control fixes the semantic weakness of textless speech language models without sacrificing the paralinguistic naturalism—backchannels, laughter, overlap, gaps—that makes them sound human. On the Fisher corpus, the authors report a relative perplexity reduction of 65.8% versus dGSLM (from 1228.82 to 421.29, approaching the ground-truth value of 371.16) and a meaningfulness MOS improvement from 1.52 to 4.08, while naturalness MOS stays comparable. The point of the work is to show that text and speech units can cooperate rather than compete in spoken dialogue generation.","feed_headline":"Phoneme conditioning cuts spoken-dialogue perplexity 65.8%","feed_subtitle":"LLM-written phoneme conditioning adds meaning to textless speech without losing natural turn-taking.","key_machinery":"The load-bearing object is the spoken phoneme sequence: a discretized, duration-carrying text representation built by forced alignment and 20 ms phoneme repetition, which lets the two-tower transformer duration predictor produce the conditioning input for dGSLM. The two-tower transformer, trained with teacher forcing, edge unit loss, and delayed duration prediction, converts written phonemes into these sequences; the conditioned dGSLM then treats the phoneme sequence as the first 4,000 tokens and continues with audio tokens, so the text determines what is said while speech units determine how it sounds. A post-processing step inserts silence tokens whenever overlaps exceed 0.6 seconds, capping overlaps at about 0.3 seconds.","core_discovery":"The authors aim to establish that a textless spoken-dialogue model can be made semantically coherent by prepending a phoneme-level textual condition derived from an LLM's output, without giving up the paralinguistic detail that makes such models natural. The conditioning signal is not plain written phonemes but spoken phoneme sequences: each phoneme is repeated once per 20 ms frame according to forced-alignment durations, with a silence phoneme added to mark utterance boundaries. During inference, a two-tower transformer predicts these sequences from written text, and dGSLM continues from them to produce speech units that HiFi-GAN decodes into audio. The reported evidence is that SLIDE-2 reaches a transcript perplexity of 421.29 versus 1228.82 for dGSLM, a relative gap of only 11.9% from ground truth, while turn-taking event statistics (IPUs, pauses, gaps, overlaps) remain close to both dGSLM and ground truth.","pith_inferences":["The paper does not test whether the same 20 ms phoneme-repetition encoding transfers to other speech-unit models; if it does, phoneme conditioning could become a general plug-in for unit-based speech generation.","One consequence the authors leave implicit is that the duration predictor, not the LLM, may be the main bottleneck: replacing forced-alignment-derived durations with a learned prosody model would test whether more natural rhythm further closes the perplexity gap to ground truth.","Because the 0.6 second overlap post-processing is applied at inference, a fair naturalism check would compare raw versus post-processed output; if silence insertion is doing most of the turn-taking work, the claim that conditioning alone preserves fluidity is weaker than it appears.","A useful extension would be to condition on discourse-level signals (topic, intent, emotion) alongside phonemes, turning SLIDE's pipeline into a controllable spoken-dialogue generator rather than just LLM-following vocalization."],"forward_implications":["SLIDE-2's transcript perplexity of 421.29 versus 1228.82 for dGSLM implies that a textless SLM can be steered semantically by phoneme conditioning while still generating speech units rather than text.","The closeness of SLIDE's turn-taking statistics to ground truth (for example, 31.3 versus 27.3 IPUs per minute, 15.8 versus 8.2 overlaps) implies that naturalistic overlap and gap behavior is not destroyed by the phoneme condition.","The M-MOS jump from 1.52 to 4.08 implies that listeners perceive LLM-generated continuations as far more meaningful, making spoken-dialogue generation viable for content-bearing interactive applications.","Because the method adds a duration predictor and conditions an existing dGSLM core, it implies an upgrade path: existing textless SLMs can be made text-controllable without redesigning their generative speech-unit core."],"supporting_citations":[{"why":"Supplies the dGSLM architecture, the two-tower transformer training scheme, and the baseline that SLIDE conditions and compares against.","marker":"[1]"},{"why":"Provides HuBERT discrete speech units used to encode audio tokens and the textless representations the model works with.","marker":"[17]"},{"why":"Provides HiFi-GAN, the units-to-speech decoder that turns generated audio tokens into waveform.","marker":"[23]"},{"why":"Provides the Fisher corpus, the 2000-hour stereo conversation dataset used for training and evaluation.","marker":"[26]"},{"why":"Provides Whisper-v3, used for transcribing prompt dialogue audio and for transcribing generated audio during semantic evaluation.","marker":"[28]"},{"why":"Provides GPT-4o, the LLM that generates the textual dialogue continuations for SLIDE-2.","marker":"[29]"},{"why":"Provides the Montreal Forced Aligner used to obtain phoneme-level durations that define the spoken phoneme sequences.","marker":"[31]"},{"why":"Provides DialoGPT, the model used to compute transcript perplexity as the semantic coherence metric.","marker":"[34]"}],"fun_headline_variants":["SLIDE: LLM phonemes cut spoken-dialogue perplexity 65.8%","Textless speech gets semantic grip via LLM phoneme planning","Spoken dialogue keeps natural turns, gains LLM meaning via SLIDE","SLIDE: Phoneme timing from LLM yields 65.8% perplexity drop","LLM-written phoneme sequences anchor SLIDE spoken dialogue"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that representing phoneme durations by repeating each phoneme every 20 ms—derived from forced alignment—preserves enough rhythm and turn-taking information that the conditioned model sounds as natural as the original textless dGSLM; if this repetition-based conditioning distorts timing or fluency, the naturalism claim fails.","fun_headline_variants_meta":{"raw":{"variants":["SLIDE: LLM phonemes cut spoken-dialogue perplexity 65.8%","Textless speech gets semantic grip via LLM phoneme planning","Spoken dialogue keeps natural turns, gains LLM meaning via SLIDE","SLIDE: Phoneme timing from LLM yields 65.8% perplexity drop","LLM-written phoneme sequences anchor SLIDE spoken dialogue"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00035,"raw_usage":{"total_tokens":1877,"prompt_tokens":876,"completion_tokens":1001,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":901}},"tokens_in":492,"tokens_out":1001,"duration_ms":9657,"temperature":1.0,"reasoning_tokens":901,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:41:17.227947+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of Fisher dialogues, run the SLIDE-2 pipeline end to end, transcribe the generated audio with Whisper, and compute DialoGPT perplexity against dGSLM output: the central claim collapses if the 65.8% relative reduction is not reproduced, or if a listener panel rates SLIDE-2 naturalness below dGSLM's 4.14 by a margin exceeding the confidence interval.","supporting_citations":[{"cited_title":"Generative spoken dialogue language modeling,","cited_arxiv_id":null,"evidence_quote":"Supplies the dGSLM architecture, the two-tower transformer training scheme, and the baseline that SLIDE conditions and compares against."},{"cited_title":"The fisher corpus: A resource for the next generations of speech-to-text","cited_arxiv_id":null,"evidence_quote":"Provides the Fisher corpus, the 2000-hour stereo conversation dataset used for training and evaluation."},{"cited_title":"Montreal forced aligner: Trainable text-speech alignment using kaldi","cited_arxiv_id":null,"evidence_quote":"Provides the Montreal Forced Aligner used to obtain phoneme-level durations that define the spoken phoneme sequences."},{"cited_title":"Dialogpt: Large-scale generative pre-training for conversational response generation,","cited_arxiv_id":null,"evidence_quote":"Provides DialoGPT, the model used to compute transcript perplexity as the semantic coherence metric."}],"review_version":1}