{"id":"a201631b-0297-436d-b528-6700f627f4b3","arxiv_id":"2608.11898","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Staged continual pre-training, dialect fine-tuning, and on-policy self-distillation improves Chinese multi-dialect ASR while preserving Mandarin CER, outperforming continued teacher-forced fine-tuning.","lead":"This paper adapts a large Mandarin ASR model to recognize Chinese dialects without losing Mandarin accuracy, using a three-stage pipeline whose final stage trains the model on its own decoded outputs. The key result is that this on-policy self-distillation improves dialect character error rate while holding Mandarin flat, where continued standard fine-tuning raises Mandarin errors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Internal speaker-overlap leakage could explain the reported gains; §4.1 only guarantees speaker holdout 'when speaker IDs are available' and never reports coverage, so the integrity of Tables 4–8 is unverified.","rationale":"The reader's weakest_assumption identifies the same data-integrity concern that I consider most load-bearing: the guarantee of speaker holdout is conditional, and for the internal data the paper does not report how often speaker IDs are available. This concern is concrete and testable, and it is the single point most likely to invalidate the central empirical claim. If speaker overlap exists, the OPSD improvements on internal dialects and the 'retained Mandarin' result could be artifacts of speaker memorization rather than the method. I agree with the reader's identification, and I add a specific verification step (speaker-overlap audit followed by re-evaluation on provably held-out speakers). The reader's CONDITIONAL verdict already accounts for this uncertainty, so I do not propose changing the verdict. Other concerns—such as the lack of confidence intervals, single runs, and unshipped artifacts—are real but secondary; they affect precision and reproducibility rather than the logical validity of the comparison. The controlled comparison in Table 8 is well designed: OPSD and Continued SFT start from the same SFT checkpoint, use the same Dref and schedule, and differ only in supervision signal. That design strengthens the paper's causal claim, but it cannot fix a leaked evaluation set. Therefore the speaker-overlap audit is the most urgent check before the claim is treated as established.","tokens_in":10946,"tokens_out":6251,"duration_ms":66539,"concrete_test":"Compute the speaker-ID overlap between the internal training and refinement partitions and the Internal Dialect test suite. Report the coverage rate (fraction of internal utterances with speaker IDs) and the number of overlapping speakers. Then re-evaluate the SFT checkpoint and the OPSD checkpoint on only those internal test utterances whose speakers are provably absent from training and refinement, and recompute the Table 8 averages. If the OPSD-vs-Continued-SFT gap on Mandarin Avg. or Dialect Avg. shifts by more than about 0.2% absolute after removing overlapping speakers, the leakage risk is material; if the numbers are unchanged, the data-integrity premise is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—OPSD lowers dialect CER without raising Mandarin CER relative to Continued SFT—rests entirely on the evaluation numbers in Tables 4–8. Every one of those numbers depends on the §4.1 premise that test speakers are held out and that utterance-level de-duplication removes overlap across training, refinement, development, and test sets. The paper explicitly weakens this premise to 'when speaker IDs are available.' For the roughly 34.1k hours of internal data, the paper never states what fraction of recordings have speaker IDs, how many distinct speakers are represented, or whether any of those speakers appear in the Internal Dialect test suite. If speaker IDs are unavailable for a substantial part of the internal data, the only safeguard is utterance-level de-duplication, which does not prevent the same speaker from appearing in different utterances across training and test. This is not a flaw in the OPSD objective itself, but a data-integrity premise on which the headline comparison depends. The large per-dialect drops after OPSD (e.g., Nanchang 18.63→15.58, Kejia 32.01→28.60) could be inflated if speakers recur. The paper's own wording flags the limitation, and the authors should be required to demonstrate that the overlap is zero or to quantify its effect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-stage adaptation pipeline for multi-dialect Mandarin ASR: continual pre-training (CPT), dialect supervised fine-tuning (SFT), and on-policy self-distillation (OPSD). In the OPSD stage, the student is trained on its own decoded prefixes while a frozen teacher, conditioned on the reference transcript as privileged context, provides soft token-level KL targets. The authors instantiate the framework with Qwen3-ASR-1.7B and evaluate on public Mandarin/dialect test sets plus an 18-dialect internal suite. The central empirical claim is that, under matched refinement data and schedule, OPSD improves dialect CER without raising Mandarin CER, whereas continued teacher-forced SFT raises Mandarin CER.","tokens_in":11200,"tokens_out":5471,"duration_ms":51788,"significance":"If the empirical claim holds, the paper makes a practical contribution: it shows that the choice of refinement objective, rather than extra data or training budget, can mitigate the dialect/Mandarin trade-off in ASR adaptation. The central comparison in Table 8 is well controlled because Continued SFT and OPSD start from the same SFT checkpoint and share the same Dref, learning rate, schedule, and decoder; the only difference is the supervision signal. The paper also provides a clear evaluation protocol with macro-averages over 31 test sets and promises release of weights and evaluation scripts, which will help reproducibility. The main uncertainties are data-integrity and statistical reliability, not the soundness of the method derivation.","major_comments":[{"comment":"The paper states that 'when speaker IDs are available, test speakers are held out from training and refinement' but never reports what fraction of the ~34.1k hours of internal data have speaker IDs, how many distinct speakers are represented, or whether any of those speakers appear in the Internal Dialect test suite. Because Tables 4, 6, and 8 all rely on these test sets, and the claimed OPSD gains include large per-dialect drops (e.g., Nanchang 18.63→15.58 and Kejia 32.01→28.60), speaker-level leakage could inflate the headline results. Please report speaker-ID coverage and overlap counts for training/refinement versus development/test, or re-evaluate the affected tables on a verified non-overlapping subset.","section":"§4.1, Tables 4–8"},{"comment":"All CER numbers are single runs with no confidence intervals or significance tests. The crucial dialect comparison is small: Dialect Avg. CER is 12.89 for Continued SFT versus 12.79 for OPSD, and several per-set differences are well under 0.5 CER. The claim that OPSD 'improves dialect recognition without raising Mandarin CER' therefore rests on differences that may be within run-to-run noise. At minimum, the refinement stage should be repeated with multiple seeds (and ideally several Dref subsamples) and the variance reported.","section":"§6.2, Table 8"},{"comment":"The objective uses forward KL D_KL(q_t || p_θ) with prefixes sampled from the student. Since the student generates the prefixes, the more common on-policy choice is reverse KL or a sample-weighted objective; the paper gives no justification for forward KL on student-generated states. This is not fatal for the empirical comparison, but it is a load-bearing design decision that should be discussed or ablated.","section":"§3.4, Eq. (3)"}],"minor_comments":[{"comment":"Report per-dialect hour counts for the internal data; a single 34.1k total makes it impossible to assess coverage of dialects such as Kejia or Nanchang, which are central to the analysis in Section 5.2.","section":"Table 1"},{"comment":"Report the number of optimizer steps for the refinement stage and the random seeds; 'one epoch' depends on the exact Dref size, and the update count is needed to confirm that Continued SFT and OPSD truly share the same training budget.","section":"§4.4"},{"comment":"The sentence 'OPSD improves all three evaluation groups after SFT' is true only at the macro-average level; a few internal dialects (e.g., Sichuan 5.13→5.38, Shanghai 11.94→12.07) regress, and this should be stated in the same paragraph for precision.","section":"§5.1"},{"comment":"Clarify whether the 13 public test sets use the official train/eval splits of their source corpora, and report any deduplication with the training collection beyond utterance-level de-duplication.","section":"§4.1"},{"comment":"The novelty relative to on-policy distillation [1] and self-distilled reasoner [37] should be stated crisply; as written, the relationship is only implicit.","section":"§2.3, §3.4"}],"recommendation":"major_revision","confidential_remarks":"The main risk is speaker overlap in the internal test suite; if the authors cannot verify the holdout, the paper's headline numbers would be compromised. This is fixable by reporting coverage and overlap statistics or re-running the affected evaluations, so I recommend major revision rather than rejection. The controlled comparison in Table 8 is a real strength, but the absence of any variance estimate makes the small reported margins hard to evaluate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is worth reading if you work on ASR adaptation or on the dialect-vs-Mandarin forgetting trade-off. The central claim is that after dialect SFT on Qwen3-ASR-1.7B, an on-policy self-distillation refinement stage (OPSD) improves both dialect and Mandarin CER, while continuing teacher-forced fine-tuning on the same refinement data raises Mandarin CER from 3.40% to 4.43%. Table 8 is the heart of the paper, and the comparison is properly controlled: both runs start from the same SFT checkpoint and use the same ~5k hours of selected hard dialect data, the same learning rate, batch size, and one epoch. The only difference is the objective. That is a real and useful result if it holds.\n\nWhat is new is the ASR-specific instantiation: a frozen teacher copy of the SFT checkpoint receives the reference transcript as privileged context and provides soft token-level targets on the student's own decoded prefixes. The authors cite on-policy distillation [1] and the self-distilled reasoner [37]; they are not claiming to invent the general idea. The contribution is the controlled demonstration for multi-dialect Mandarin ASR. The writing is honest: they report SpeechIO-1 not returning to the CPT value and a few dialects where SFT beats OPSD.\n\nThe soft spots are in data integrity and statistical reporting, not in the method's logic. Section 4.1 says test speakers are held out 'when speaker IDs are available.' For the roughly 34.1k hours of internal data, the paper never states how much of that data has speaker IDs, how many distinct speakers there are, or whether internal test speakers overlap training or refinement. Utterance-level de-duplication does not prevent the same speaker appearing in different utterances in training and test. If speakers recur, the large OPSD gains on Kejia (32.01 to 28.60) and Nanchang (18.63 to 15.58) could be partly leakage. This is the main thing I would want fixed before trusting the numbers.\n\nAlso, every CER is a single run. No confidence intervals, no significance tests, and no code or weights are shipped yet. The differences in Table 8 are directionally consistent across many test sets, which helps, but some individual Mandarin gains are small. I would not call the central claim suspect; it is plausible. I would call it unverified until speaker-ID coverage is reported and repeated seeds or leakage checks appear.\n\nWho is this for? ASR researchers and engineers adapting models to dialect or long-tail speech, and people working on continual learning for speech. It deserves a serious referee. The comparison is well designed and the result would be practically useful, but the authors should be required to demonstrate test-speaker disjointness and to report variance before the claim is treated as established.","headline":"A genuinely well-controlled comparison showing on-policy self-distillation beats continued fine-tuning for multi-dialect ASR refinement, with the main caveat being unverified speaker-disjointness and single-run numbers.","tokens_in":11736,"tokens_out":3456,"would_cite":true,"duration_ms":31594,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that on-policy self-distillation improves dialect recognition in autoregressive Mandarin ASR while keeping Mandarin character error rate flat, and that continued teacher-forced cross-entropy on the same data raises it.","keywords":["Chinese multi-dialect ASR","dialect adaptation","on-policy self-distillation","self-distillation","continual pre-training","supervised fine-tuning","character error rate","autoregressive ASR"],"falsifier":"Run an exact and approximate duplicate-audio or duplicate-transcript search between the 5k-hour refinement set and the 13 public test sets; finding any test utterance in the refinement data would mean the OPSD gains could come from memorization rather than from the distillation objective.","tokens_in":10746,"feed_emoji":"🗣️","tokens_out":6787,"duration_ms":62306,"temperature":0.7,"pith_summary":"The paper asks how to adapt a capable Mandarin ASR system to Chinese dialects without paying the usual price of forgetting Mandarin. It proposes a three-stage recipe—continual pre-training, dialect supervised fine-tuning, and a final on-policy self-distillation (OPSD) step—and instantiates it with a 1.7-billion-parameter open-source ASR model. The central finding is that, under matched refinement data and schedule, OPSD improves dialect character error rate (CER) while leaving Mandarin CER flat or slightly better, whereas continuing standard cross-entropy fine-tuning on the same data pushes Mandarin CER up. A sympathetic reader would care because the result separates the adaptation signal from the data and schedule, suggesting that the training objective, not just the data, controls the Mandarin–dialect trade-off.","feed_headline":"Distill on decoded speech: dialect errors fall, Mandarin holds","feed_subtitle":"A student ASR model learns from its own decoded prefixes while a frozen teacher keeps Mandarin error rates flat.","key_machinery":"The load-bearing mechanism is the OPSD objective applied to autoregressive ASR. For each utterance, the student first decodes its own prediction $\\hat{y}_{\\le T}$ from the audio; at each position $t$, the frozen teacher $\\bar{\\theta}$ computes a distribution $q_t(v) = p_{\\bar{\\theta}}(v \\mid x, c(y^\\star), \\hat{y}_{<t})$ over the vocabulary, where $c(y^\\star)$ is the reference transcript given only to the teacher as a privileged prompt; the student's gradient is $\\sum_t D_{KL}(q_t \\| p_\\theta(\\cdot \\mid x, \\hat{y}_{<t}))$, with no auxiliary cross-entropy term. This combines two things: on-policy states (the student learns from its own decoded prefixes rather than gold prefixes) and soft supervision (the teacher's distribution instead of a one-hot reference), which together address the train–test mismatch and soften the dialect update.","core_discovery":"On the paper's own terms, the discovery is that replacing hard cross-entropy updates with on-policy self-distillation in the final refinement stage lets an autoregressive ASR model learn more from dialect speech without overwriting its Mandarin ability. The student samples its own hypothesis for each training utterance; a frozen teacher—initialized from the same SFT checkpoint and given the reference transcript as privileged context—supplies soft token-level targets on those student-decoded prefixes; the student minimizes the KL divergence to those targets. With the same 5k-hour refinement set, same batch size, and same one-epoch schedule, OPSD lowers the Mandarin average CER from 3.40% to 3.27% and the dialect average from 13.16% to 12.79%, while Continued SFT raises the Mandarin average to 4.43%. The paper presents this as evidence that the objective, not extra data or training budget, is responsible for the improved Mandarin–dialect balance.","pith_inferences":["Beyond what the paper tests, the same on-policy self-distillation recipe should transfer to other autoregressive speech tasks—such as dialect text-to-speech or code-switched ASR—because the train–test mismatch it addresses is not specific to Chinese.","The reference-conditioned teacher acts as an oracle at training time; an interesting extension would be to vary the teacher's privileged context or use multiple teachers to push harder on long-tail dialects such as Kejia and Chaoshan.","The paper isolates the objective but not the data-selection policy (high-CER utterances, capped per dialect); the interaction between this curation strategy and OPSD is untested and could be probed by swapping the refinement set.","A testable hypothesis implied but not stated is that OPSD's Mandarin retention depends on the teacher being initialized from the SFT checkpoint; a teacher initialized from the CPT checkpoint might behave differently."],"forward_implications":["On matched refinement data, OPSD yields lower CER than continued teacher-forced SFT on all four macro-averages (Mandarin, Dialect, Internal, and Overall), and it lowers Mandarin Avg. to 3.27% while Continued SFT raises it to 4.43%.","The full staged pipeline (CPT, then dialect SFT, then OPSD) lowers Mandarin Avg. from 3.46% to 3.27%, Dialect Avg. from 15.37% to 12.79%, and Internal Avg. from 21.01% to 12.42%.","Because the only difference between OPSD and Continued SFT is the supervision signal, the gain comes from distillation on on-policy prefixes rather than from extra data or a different training schedule.","OPSD is not uniformly best on every test set—SpeechIO-1 and a few dialects such as Shanghai and Sichuan are slightly better under earlier stages—so the appropriate use is as a final refinement on dialects where early-error accumulation dominates.","Direct dialect SFT without CPT substantially raises Mandarin Avg. to 5.16%, so CPT before SFT is needed for a stable Mandarin–dialect trade-off in this pipeline."],"supporting_citations":[{"why":"Original on-policy self-distillation idea this paper adapts; supplies the teacher-student distilled-loss formulation.","marker":"[37]"},{"why":"Technical report for Qwen3-ASR, the base model and starting checkpoint of the pipeline.","marker":"[25]"},{"why":"Establishes on-policy distillation for language models, the conceptual basis for training on student-decoded prefixes with dense teacher feedback.","marker":"[1]"},{"why":"Documents the train–test mismatch in sequence models that OPSD is designed to close.","marker":"[6]"},{"why":"Cantonese corpus used in continual pre-training and as a dialect evaluation set.","marker":"[18]"},{"why":"Sichuan dialect corpus used for training and evaluation.","marker":"[11]"},{"why":"Wu dialect corpus used for training and evaluation.","marker":"[27]"},{"why":"Large Mandarin corpus underpinning the Mandarin side of the training mixture and test sets.","marker":"[35]"},{"why":"Mandarin-plus-subdialect corpus used as training data and as an accented-Mandarin evaluation set.","marker":"[24]"}],"fun_headline_variants":["Distill on own decoded prefixes: dialect errors drop, Mandarin holds","OPSD: dialect accuracy up, Mandarin error rate flat","Self-distillation on student-decoded speech: better dialects, same Mandarin","Distill from frozen teacher: dialect down, Mandarin unchanged"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes that no development or test utterance appears among the roughly 100k hours of training or the 5k-hour refinement data, with speaker-ID holdout and utterance de-duplication applied only where speaker IDs are available.","fun_headline_variants_meta":{"raw":{"variants":["Distill on own decoded prefixes: dialect errors drop, Mandarin holds","OPSD: dialect accuracy up, Mandarin error rate flat","Self-distillation on student-decoded speech: better dialects, same Mandarin","Distill from frozen teacher: dialect down, Mandarin unchanged"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000544,"raw_usage":{"total_tokens":2617,"prompt_tokens":975,"completion_tokens":1642,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":1569}},"tokens_in":591,"tokens_out":1642,"duration_ms":11039,"temperature":1.0,"reasoning_tokens":1569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:22:35.890278+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an exact and approximate duplicate-audio or duplicate-transcript search between the 5k-hour refinement set and the 13 public test sets; finding any test utterance in the refinement data would mean the OPSD gains could come from memorization rather than from the distillation objective.","supporting_citations":[{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Establishes on-policy distillation for language models, the conceptual basis for training on student-decoded prefixes with dense teacher feedback."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Documents the train–test mismatch in sequence models that OPSD is designed to close."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Cantonese corpus used in continual pre-training and as a dialect evaluation set."},{"cited_title":"In: ACL Findings","cited_arxiv_id":null,"evidence_quote":"Wu dialect corpus used for training and evaluation."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Large Mandarin corpus underpinning the Mandarin side of the training mixture and test sets."},{"cited_title":"In: NeurIPS Datasets and Benchmarks (2021)","cited_arxiv_id":null,"evidence_quote":"Mandarin-plus-subdialect corpus used as training data and as an accented-Mandarin evaluation set."}],"review_version":1}