{"id":"fa22f6f3-98fb-4bf0-8a25-0e6da03ae69f","arxiv_id":"2608.08067","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DialectS2S adapts an end-to-end speech dialogue model to Chinese dialects by synthesizing dialect training data and adding a self-aligned speech supervision stage that uses the model's own text predictions to build TTS targets.","lead":"The paper introduces DialectS2S, an end-to-end speech dialogue model trained for three Chinese dialects using LLM-rewritten dialogue texts and TTS-synthesized dialect speech. Its main innovation is a two-stage post-training strategy where the model's own text predictions are used to generate aligned speech supervision, and the authors report gains in dialect matching, response quality, and intelligibility.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Speech intelligibility claim compares DialectS2S CER to an external ASR benchmark average, not to baseline models under the same protocol; the claimed significant intelligibility advantage over baselines is not established.","rationale":"The reader's weakest_assumption precisely identifies the load-bearing weakness: the intelligibility claim is based on a CER metric computed with the model's own text as reference and compared against an external ASR benchmark average rather than against the same baselines on the same generated speech. My stress-test confirms this is the most significant gap in the paper's central claim. The central claim has three pillars; dialect consistency and response quality are evaluated against baselines on the same benchmark, but intelligibility is not. Because the abstract and conclusion explicitly claim \"substantial improvements in speech intelligibility\" over open-source baselines, the broken comparison directly undermines a headline result. This concern does not invalidate the method itself or the other two evaluation pillars; it raises the required standard of evidence. The CONDITIONAL verdict is therefore appropriate: the paper should be accepted only if the intelligibility claim is re-evaluated with a like-for-like baseline comparison or human listening tests. No change to the reader's verdict is needed.","tokens_in":8540,"tokens_out":8738,"duration_ms":87065,"concrete_test":"Run the same CER protocol on all baselines: for each model (GLM-4-Voice, Qwen2.5-Omni, Kimi-Audio, Step-Audio2, OpenS2S, MiniCPM-o-4.5), generate responses to the same 250-item multi-dialect benchmark, transcribe the generated speech with FRASR2-AED, and compute CER against that model's own synchronously generated text. Report per-dialect CER and averages. If DialectS2S's dialect CER (Table 3) is not below the best baseline's CER under this identical protocol, the claimed intelligibility improvement over baselines is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim includes \"substantial improvements in speech intelligibility\" over open-source baselines. The only evidence for this is Section 5.4 and Table 3, where DialectS2S's generated speech is transcribed with FRASR2-AED and compared with that ASR system's published average CER on a natural-speech dialect benchmark. This is not a like-for-like comparison: (i) the baseline models are not run through the same CER pipeline on the same test queries, so no relative intelligibility advantage over baselines is demonstrated; (ii) the reference text for DialectS2S is the model's own synchronously generated text, which is not comparable to the benchmark's reference transcripts; (iii) the speech domain (synthetic TTS output vs. natural human speech) and the test-set distribution both differ. Consequently, the conclusion that DialectS2S \"significantly outperforms\" baselines in intelligibility is unsupported by the reported numbers. The self-aligned supervision ablation in Table 5 uses a consistent self-reference protocol across conditions, so it provides some internal evidence, but it still lacks error bars or significance tests and does not involve any of the named baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DialectS2S, an end-to-end speech dialogue model for low-resource Chinese dialects built on OpenS2S. It introduces a dialect speech dialogue synthesis pipeline (LLM-based text rewriting, CosyVoice2 TTS, UTMOS filtering) and a two-stage post-training procedure: mixed-data supervised fine-tuning followed by self-aligned speech supervision, in which the fine-tuned Thinker generates text that is synthesized via a dialect TTS and used as the Talker training target. The model is evaluated on a 250-sample multilingual/dialect benchmark for language matching accuracy, response quality (via an LLM judge), and speech intelligibility (via CER), compared with six open-source baselines, with additional ablations varying SFT epochs and the self-aligned stage.","tokens_in":8699,"tokens_out":4850,"duration_ms":45768,"significance":"If the results were validated, the contributions would be practically valuable: the authors open-source checkpoints, datasets, and code; the synthesis pipeline offers a scalable way to create dialect speech dialogue data; and the self-aligned supervision idea is an interesting approach to adapting speech supervision to evolving hidden representations. The paper also provides a useful comparison of several open-source speech dialogue models on dialect inputs. However, the central intelligibility claim is currently undermined by the evaluation protocol, which does not compare against baselines under the same conditions and uses the model's own text as the CER reference, making the claimed 'substantial improvements in speech intelligibility' unsupported. The underlying idea remains plausible, and the gaps could be addressed with additional experiments.","major_comments":[{"comment":"The claim that DialectS2S 'significantly outperforms existing open-source baselines' in speech intelligibility is not supported by the reported data. Table 3 compares DialectS2S's CER (8.69% dialect average) to the FRASR2-AED benchmark average (11.67%) on natural speech, not to the CER of any baseline model on the same generated speech. None of the baseline models are run through the same ASR pipeline on the same test queries, and the reference for DialectS2S is its own synchronously generated text rather than a human transcript. This is not a like-for-like comparison and cannot establish a relative intelligibility advantage over baselines.","section":"§5.4, Table 3"},{"comment":"The CER evaluation uses the model's own generated text as the reference for its own speech. Because the self-aligned supervision training synthesizes speech from the model's own text predictions (Eq. 3) and trains the Talker to produce speech matching that text (Eq. 4), the CER reductions in Table 5 may simply reflect improved internal text-speech consistency, which is exactly what the training optimizes, rather than improved absolute intelligibility. Human listening tests, or an external ASR evaluated against fixed human reference transcripts, are needed to establish intelligibility gains; the current metric conflates self-consistency with intelligibility.","section":"§5.4, Table 5, and §3.3 Eqs. (2)-(4)"},{"comment":"The self-aligned supervision is circular in that the training target is derived from the model's own predictions. If the Thinker produces erroneous text, the TTS synthesizes speech for that erroneous text and the Talker is trained to match it, potentially reinforcing the model's own errors. The claimed mechanism—reducing semantic mismatch between hidden states and speech targets—is plausible, but the reported CER results do not separate the effect of aligning with the model's own semantics from the effect of making the model more self-consistent. The authors should report CER against human reference transcripts, or compare with a condition where supervision is generated from ground-truth text, to support the mechanism claim.","section":"§3.3, Eqs. (2)-(4)"},{"comment":"The language-matching and response-quality evaluations rely on a single 250-sample benchmark, automatic language identification (FireRedLID), and an LLM judge (Qwen3-Plus) without reported statistical significance, confidence intervals, or human validation. The near-zero baseline scores for dialect conditions (e.g., 0-2% for Tianjin) are expected because those models are not designed for dialects, but the claim of 'significant' improvements over baselines is not supported without significance tests or a more thorough analysis of the judge's reliability.","section":"§5.1-§5.3"}],"minor_comments":[{"comment":"The text states that 'Further seed-set details are provided in Appendix A,' but Appendix A does not include details on the 100 dialect seed utterances or the UTMOS filtering threshold; these would be needed to reproduce the data pipeline.","section":"§3.2, Appendix A"},{"comment":"The cascade comparison is based on only 50 Sichuanese samples, which is a very small sample size; the latency and quality numbers should be interpreted with caution, and the paper should state whether these differences are stable across repeated runs.","section":"Appendix A, Table 6"},{"comment":"The PCA visualization of 'final-layer Thinker hidden states' does not specify which layer is used, how token-level representations are aggregated, or which samples are projected; the reader cannot verify the claim of hidden-state drift without these details.","section":"Figure 2"},{"comment":"The paper states that sft-3ep and DialectS2S are trained with comparable numbers of optimization steps, but no step counts are given; reporting them would strengthen the fairness of the ablation comparison.","section":"§6"},{"comment":"The sentence 'Previous experiments show that existing baseline models mainly generate Mandarin Chinese rather than dialect speech under dialect interaction settings' is vague; it should reference the specific tables (e.g., Table 1) or provide the evidence explicitly.","section":"§5.4"},{"comment":"The paper claims to 'fully open-source' the framework but does not provide a repository link or URL; the authors should include an accessible link in the final version.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The major issue is the evaluation protocol for the headline intelligibility claim. The comparison to an external benchmark average and the use of self-references in CER are not acceptable evidence for outperforming baselines; this needs to be fixed with proper baseline comparisons and human or external-reference evaluation. The self-aligned training idea is interesting and the data pipeline is useful, so the paper is salvageable, but the current experimental evidence does not support the strongest claims. I would also recommend checking whether the language matching results for dialect conditions are inflated by the choice of LID and whether the LLM judge has a bias toward the model's own dialectal text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the quick read on DialectS2S. It's a plausible, clearly written contribution to end-to-end speech dialogue for low-resource Chinese dialects. The genuinely new piece is the self-aligned speech supervision stage: after SFT, you synthesize speech targets from the model's own text predictions, so the TTS supervision tracks the model's evolved hidden states. That's a real twist, and the ablation in Table 5 supports it: under a consistent self-reference CER protocol, DialectS2S beats both sft-2ep and sft-3ep across all dialects, and sft-3ep doesn't show the same gains. The data pipeline itself is fairly standard (LLM rewrite, TTS, UTMOS filter), but the extension to dialect dialogue with speaker-consistent reference synthesis is useful.\n\nThe soft spots are mostly in the evaluation. The headline claim that DialectS2S significantly outperforms baselines in speech intelligibility is not supported by Table 3. The comparison is against the FRASR2-AED benchmark's average CER, not against the same baselines under the same protocol; the reference text is the model's own generated text; and the domain differs (synthetic vs. natural speech). The paper actually admits that baselines mostly output Mandarin in dialect settings, which explains why no baseline CER is reported, but then the conclusion should be framed as 'intelligible dialect speech' rather than 'outperforms baselines.' That is a real overclaim.\n\nSecond, the self-aligned training has a mild circularity: targets come from the model's own text, and CER is measured against that same text. The ablation partially mitigates this because all conditions use the same protocol, but it's not a human-intelligibility measure. No error bars, no significance tests, and only 250 test samples.\n\nThird, the paper cites EchoX, which is conceptually close, and never compares to it. That's a gap worth addressing. Also, the promised checkpoints/data/code have no link or repo in the text, which matters because open source is an explicit contribution.\n\nOverall, the method is credible, the writing is honest about some limitations, and the direction is useful for the low-resource dialect community. It deserves a serious referee, but the evaluation needs major revision: run baselines through the same CER pipeline, add listening tests, and report variance. I'd send it to review with a request for those changes.","headline":"Useful new framework for end-to-end dialect speech dialogue with a genuinely novel self-aligned supervision loop, but the headline intelligibility claim is not established by the reported CER comparisons.","tokens_in":9295,"tokens_out":2055,"would_cite":true,"duration_ms":18849,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DialectS2S claims that re-synthesizing speech supervision from the model's own predicted text removes the semantic mismatch that degrades dialect speech, letting an end-to-end dialogue model outperform open baselines across three Chinese…","keywords":["end-to-end speech dialogue","Chinese dialects","low-resource speech","self-aligned speech supervision","speech-to-speech modeling","dialect data synthesis","speech intelligibility"],"falsifier":"Score DialectS2S and each open baseline with the same automatic speech recognizer on their generated speech, using each model's own text as reference and also human transcripts, and compare all systems under identical conditions; if DialectS2S does not beat the baselines on matched CER or on human listening ratings, the claimed intelligibility advantage is refuted.","tokens_in":8298,"feed_emoji":"🗣️","tokens_out":9487,"duration_ms":87604,"temperature":0.7,"pith_summary":"The paper tries to establish that end-to-end speech dialogue can be extended to low-resource Chinese dialects without the usual drop in speech quality, and that the key obstacle is a mismatch between the model's evolving semantic representations and the fixed speech targets used in training. As a model adapts to a dialect, its internal semantic space shifts; if the speech supervision stays frozen, the speech-generation module has to recover semantics it should not need to recover, and output becomes unstable and unnatural. The authors' fix is to use the adapted model's own text predictions to synthesize new speech supervision, so the targets and the hidden representations move together. They report that the resulting system, built from a data-synthesis pipeline and two-stage post-training, outperforms open baselines in dialect consistency, response quality, and intelligibility for Sichuanese, Cantonese, and the Tianjin dialect, while keeping Mandarin and English performance. A sympathetic reader would care because it offers a practical, open recipe for a capability that currently barely exists in end-to-end systems.","feed_headline":"Dialect speech dialogue beats baselines on three Chinese dialects","feed_subtitle":"A training trick aligns speech targets to the model's evolving semantics, keeping dialect responses intelligible","key_machinery":"The load-bearing mechanism is self-aligned speech supervision, where the Thinker (the module that reads speech and produces hidden semantic states $H$ and text tokens) generates text predictions $\\hat{T} = \\mathrm{Thinker}(X)$ after supervised fine-tuning, and a dialect TTS synthesizes $\\hat{U} = \\mathrm{TTS}(\\hat{T})$; the Talker (the module that maps hidden states to speech tokens) is then trained with the loss $\\mathcal{L}_{\\mathrm{AlignTalker}} = -\\sum_j \\log P(\\hat{u}_j \\mid H, \\hat{U}_{<j}; \\theta_{\\mathrm{Talker}})$. This makes the speech target track the evolving semantic space instead of remaining frozen, so the Talker models prosody and pronunciation rather than performing implicit semantic recovery. A second supporting mechanism is the data pipeline that rewrites existing Mandarin dialogue text into dialectal expressions with an LLM, synthesizes query and response speech from seed utterances, and filters samples with a naturalness scorer.","core_discovery":"The central discovery is that supervised fine-tuning changes the hidden representations of a Thinker-Talker speech dialogue model, but conventional speech supervision stays anchored to the old semantic space; the speech module then does double duty, recovering meaning while generating acoustics, and dialect speech degrades. The paper's solution is self-aligned speech supervision: after a first stage of mixed-data supervised fine-tuning, the fine-tuned Thinker generates text predictions for each input, a dialect TTS turns those predictions into speech, and that generated speech replaces the original target in the Talker's training loss. Aligning the target's semantic content with the model's current predictions reduces the mismatch, letting the Talker focus on pronunciation, prosody, and naturalness. The authors show that this second stage improves character error rate relative both to stopping after two epochs and to adding a third epoch of ordinary fine-tuning, with matched optimization steps, and that the full model beats several open speech dialogue baselines on language matching, judged response quality, and intelligibility across three Chinese dialects.","pith_inferences":["Inference: The same self-alignment idea should transfer to any speech-to-speech distribution shift, such as accented English, child speech, code-switching, or a new speaker's voice, because the underlying problem is frozen supervision lagging behind drifting hidden representations, not dialect-specific.","Inference: The reported intelligibility numbers should be read cautiously until a matched comparison exists; the paper's CER table compares the model's own generated speech against a benchmark average, so it does not yet show that the model is more intelligible than the baselines on the same utterances.","Inference: A direct mechanistic test is available: measure the distance between the final-layer Thinker hidden states and the speech token targets before and after self-aligned training; the mechanism predicts this distance shrinks even when ordinary extra fine-tuning does not.","Inference: The pipeline's ceiling for dialect authenticity is set by the LLM's dialect rewriting and the TTS's dialect synthesis; if those components produce a shallow or standardized version of a dialect, self-aligned supervision will faithfully preserve that version rather than recover the true dialect."],"forward_implications":["A pretrained end-to-end speech dialogue model can be adapted to a new low-resource dialect with only a few thousand synthesized speech pairs, without sacrificing its original Mandarin and English ability.","Adding more ordinary supervised fine-tuning does not fix speech degradation and can make it worse; the self-aligned second stage gives the intelligibility gain at matched optimization steps.","Because the whole framework, including checkpoints, datasets, and fine-tuning code, is released, other teams can reproduce the results and port the pipeline to additional dialects.","Dialect speakers can hold spoken conversations in their own variety: the model matches the input dialect in over 90% of test cases for Sichuanese, Cantonese, and Tianjin, whereas leading open baselines mostly fall back to Mandarin."],"supporting_citations":[{"why":"Supplies the base speech dialogue model and the source Mandarin dialogue corpus that the pipeline rewrites into dialects.","marker":"[17]"},{"why":"Provides the dialect TTS used for query-speech synthesis and for generating self-aligned supervision from predicted text.","marker":"[7]"},{"why":"Supplies seed dialect utterances used as reference audio for synthesizing query speech.","marker":"[15]"},{"why":"Supplies Cantonese seed utterances for query-speech synthesis.","marker":"[9]"},{"why":"Supplies Sichuanese seed utterances for query-speech synthesis.","marker":"[4]"},{"why":"Provides the LLM that rewrites dialogue text into dialectal expressions and the evaluator that scores response quality.","marker":"[22]"},{"why":"Filters synthesized speech samples by predicted naturalness to keep training data clean.","marker":"[13]"},{"why":"Supplies language identification for matching accuracy and the FRASR2-AED benchmark used for CER intelligibility.","marker":"[21]"},{"why":"Documents that supervised fine-tuning can degrade speech quality and stability, motivating the self-alignment objective.","marker":"[24]"}],"fun_headline_variants":["Self-aligned speech training boosts dialect dialogue quality","Dialect dialogue model uses self-aligned speech to stay intelligible","New trick keeps dialect speech dialogue natural and consistent","Training speech targets to match model semantics improves dialect output"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the speech-recognition error rate used to measure intelligibility reflects what human listeners actually hear, because the model is compared with an external benchmark average instead of with the same baselines on the same generated speech.","fun_headline_variants_meta":{"raw":{"variants":["Self-aligned speech training boosts dialect dialogue quality","Dialect dialogue model uses self-aligned speech to stay intelligible","New trick keeps dialect speech dialogue natural and consistent","Training speech targets to match model semantics improves dialect output"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1370,"prompt_tokens":968,"completion_tokens":402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":350}},"tokens_in":584,"tokens_out":402,"duration_ms":4517,"temperature":1.0,"reasoning_tokens":350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:27:54.175852+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score DialectS2S and each open baseline with the same automatic speech recognizer on their generated speech, using each model's own text as reference and also human transcripts, and compare all systems under identical conditions; if DialectS2S does not beat the baselines on matched CER or on human listening ratings, the claimed intelligibility advantage is refuted.","supporting_citations":[{"cited_title":"In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","cited_arxiv_id":null,"evidence_quote":"Supplies the base speech dialogue model and the source Mandarin dialogue corpus that the pipeline rewrites into dialects."},{"cited_title":"Shu et al","cited_arxiv_id":null,"evidence_quote":"Supplies seed dialect utterances used as reference audio for synthesizing query speech."},{"cited_title":"Proceedings of the AAAI Conference on Artificial Intelligence40, 31627– 31635 (Mar 2026).https://doi.org/10.1609/aaai.v40i37.40429","cited_arxiv_id":null,"evidence_quote":"Supplies Cantonese seed utterances for query-speech synthesis."},{"cited_title":"EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMs","cited_arxiv_id":"2509.09174","evidence_quote":"Documents that supervised fine-tuning can degrade speech quality and stability, motivating the self-alignment objective."}],"review_version":1}