REVIEW 4 major objections 4 minor 25 references
On the MLC-SLM 2026 two-speaker task, a three-stage adaptation recipe reduces Qwen3-ASR-1.7B's average tcpMER from 30.53 on dev to 23.70, and to 17.97 on eval.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 07:54 UTC pith:R5ZPPT66
load-bearing objection Useful challenge-system report whose headline number is internally consistent but measured on a local scoring pipeline; the adaptation gain likely holds, but the abstract overstates it as an official score. the 4 major comments →
Diarization-Guided Qwen-ASR Adaptation for Multilingual Two-Speaker Conversational Speech
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that Qwen3-ASR-1.7B, at 30.53 average tcpMER on the MLC-SLM 2026 dev set under a shared diarization output, can be adapted by a three-stage recipe to 23.70: full supervised fine-tuning on official data, LoRA on 234,333 synthetic TTS items, and GRPO reinforcement learning with a rule-based reward. The ablation isolates the contributions: full SFT delivers 6.23 of the 6.83-point gain, synthetic-speech LoRA adds 0.52 (Urdu 102.84→19.77, Tagalog 49.49→33.90), and RL adds 0.08 by stabilizing outputs. The authors read this as SFT aligning the model to the challenge's two-speaker conversational style, synthetic speech broadening hard-language coverage, and RL acting as
What carries the argument
The central machinery is the three-stage adaptation recipe applied to the Qwen3-ASR-1.7B speech language model: (1) full supervised fine-tuning (SFT) on the official training set, (2) low-rank adaptation (LoRA) on synthetic speech generated by a three-pipeline TTS framework, and (3) Group Relative Policy Optimization (GRPO), a critic-free reinforcement-learning update, with a rule-based reward combining word/character error rate with penalties for hallucination, repetition, and length deviation. The diarization front end—FSMN-VAD, CAMPPlus speaker embeddings, and spectral clustering fixed to two speakers, followed by RTTM-based cutting—turns the recording into speaker-attributed segments, bu
Load-bearing premise
The load-bearing premise is that the authors' local implementation of tcpMER, which they use for the development set and explicitly label 'our local scoring pipeline' (§1, §4.1), reproduces the official scoring protocol; if the normalization, permutation search, or per-language averaging differs, every dev-set number changes, leaving the official eval result (17.97) as the only fully validated figure.
What would settle it
Run the submitted system through the official MLC-SLM 2026 judge on the development set and compare the returned tcpMER with the reported 23.70; agreement would validate the local pipeline, while a material discrepancy would invalidate the dev-set results and the ablation decomposition.
If this is right
- The adapted system beats both official ASR baselines under the same diarization-aware protocol: Whisper-large-v3 averages 29.94 tcpMER and Omniasr-LLM-7B-v2 averages 31.52, versus the submitted system's 23.70 on dev and 17.97 on eval.
- Full SFT accounts for 6.23 of the 6.83 total dev-set gain, so for challenge-style adaptation of speech LLMs, high-quality supervised data is the dominant lever; synthetic LoRA and RL are refinements.
- The largest per-language gains are in the weakest languages—Urdu 102.84→19.77, Tagalog 49.49→33.90—indicating the recipe specifically repairs under-represented acoustic-linguistic confusions.
- GRPO with rule-based rewards (WER/CER plus hallucination, repetition, and length penalties) suppresses pathological outputs on long, noisy segments, offering a stable, human-feedback-free RL recipe for ASR.
- Because the same diarization output is shared across all ablation runs, the measured improvement is attributable to recognizer adaptation alone, not to diarization changes.
Where Pith is reading between the lines
- The paper does not explain the 5.7-point gap between its local dev score (23.70) and the official eval score (17.97); a plausible reading is that the local tcpMER implementation differs systematically from the official judge, so the dev-set ablation numbers should be treated as internally consistent rather than directly comparable to official results.
- The synthetic-data count mixes dialogue groups (166,633) and utterances (67,700); the effective LoRA training set is roughly 67,700 utterance-level samples plus the dialogue groups, so the marginal contribution of each TTS pipeline remains untested.
- A natural next experiment is to apply the same SFT+LoRA+GRPO recipe to other speech LLMs (e.g., Whisper-large-v3 or Omniasr-LLM-7B-v2) to see whether the gains are model-specific; the paper tests only Qwen3-ASR-1.7B.
- Since SFT dominates, replacing the LoRA stage with an equal-sized SFT continuation on the synthetic data might produce a different ablation conclusion; the current design conflates adaptation method with data source.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes SQZ-Qwen-ASR-1.7B, a submission to MLC-SLM 2026 Challenge Task 1. The system couples a modular diarization front end (FSMN-VAD, CAMPPlus embeddings, spectral clustering with speaker count fixed to two, RTTM-based segmentation) with Qwen3-ASR-1.7B adapted in three stages: full SFT on official training data, LoRA on TTS-synthesized speech (234,333 items from three pipelines), and GRPO reinforcement learning with WER/CER-based rewards and hallucination/repetition/length penalties. On the development set the authors report an average tcpMER of 23.70 in their local scoring pipeline, compared with 30.53 for unadapted Qwen-ASR-1.7B, a 6.83-point absolute gain; on the evaluation set they report 17.97 according to the task leaderboard. Ablations show full SFT provides the largest gain (30.53 to 24.3), synthetic-speech LoRA adds 0.52, and RL adds 0.08.
Significance. If the local scoring implementation faithfully matches the official tcpMER protocol, the paper provides a clean and largely reproducible recipe whose main message (SFT on challenge data dominates; synthetic speech and RL give smaller refinements) is useful for the MLC-SLM task and for multilingual conversational ASR generally. The diarization front end and training hyperparameters are specified in unusual detail, and the per-language Table 2 allows independent checks. I do not see a circularity problem: the claim is an external benchmark measurement, not a derived quantity. The central risk is measurement validity, not internal inconsistency.
major comments (4)
- [Abstract; §1; §4.1; Table 2] The headline dev-set number conflates two scoring pipelines. The abstract says 'On the official development set ... 23.70', but §1 states this is 'in our local scoring pipeline' and the Table 2 caption says 'based in our local pipeline'. The 6.83-point gain (30.53 to 23.70), all ablations in Table 3, and the comparisons to Whisper/Omniasr are local; the eval number 17.97 is official/leaderboard. No evidence is given that the local tcpMER implementation (normalization, permutation search, per-language WER/CER selection, 21-condition averaging) matches the official judge. Please provide a direct local-vs-official comparison on the dev set, or report the official dev score if available, and clearly state which numbers are comparable. Without this, the abstract overstates the result as an official benchmark.
- [Table 2 footnote/caption] The caption says all systems are evaluated 'based in our local pipeline', but the footnote says Whisper-large-v3 and Omniasr-LLM-7B-v2 are 'evaluated under the contest protocol'. If these baseline columns are produced by the official judge while Qwen-ASR-1.7B and SQZ-Qwen-ASR-1.7B use the local pipeline, the columns are not comparable. If they are all local, the footnote is misleading. Because the local pipeline may be stricter or looser than the official protocol, cross-column absolute differences could be artifacts. Please run every system through one identical scoring implementation and state this explicitly.
- [§4.2; Table 3] All results are single-run numbers with no confidence intervals, significance tests, or segment/condition-level variability. The reported LoRA gain is 0.52 and the RL gain is 0.08; without some uncertainty estimate (e.g., bootstrap over the 21 language conditions, multiple decoding runs, or paired tests), these increments are not distinguishable from noise. Since §4.2 uses these small increments to support claims about 'consistent further reductions' and RL as a 'low-risk stabilizer', the absence of any variance measure is a load-bearing gap.
- [§4.1; §5] The dev-set score (23.70 local) differs from the eval-set score (17.97 official) by 5.7 points. If both numbers were produced under the same protocol, this large gap would suggest a substantial domain shift or model-selection effect; if they are from different pipelines, the gap is partly a scoring artifact. The paper does not discuss this discrepancy at all. Please explain the relationship between the two numbers, including whether the dev set is harder than the eval set, whether hyperparameters were selected on dev, or whether local and official scorings differ.
minor comments (4)
- [Abstract; §5] The abstract should carry the same qualification as §5: the dev number is 'in our local scoring pipeline'. Currently the abstract implies the 23.70 is an official dev result, which is contradicted by the body.
- [Table 3] Use consistent decimal precision: 24.3, 23.78, 23.70. The one-decimal 24.3 makes the LoRA increment look less precise than the others.
- [§2.2] The notation 'min/max/oracle num_spks=2' is unclear. State plainly whether the speaker count is always fixed to two or whether an oracle count is used for that step; the challenge setting suggests the former.
- [Table 2] The Urdu Qwen-ASR-1.7B value of 102.84 exceeds 100; a one-sentence explanation (e.g., tcpWER can exceed 100 after permutation/insertion penalties) would help readers interpret extreme values.
Circularity Check
No circularity found: the system is an empirical benchmark submission with an external leaderboard check, not a derivation that reduces to its own inputs.
full rationale
The paper reports an empirical SD+ASR system for the MLC-SLM 2026 challenge: a diarization front end plus Qwen3-ASR-1.7B adapted by SFT, LoRA on synthetic speech, and GRPO. The headline numbers are benchmark measurements against an external challenge protocol, not quantities derived from definitions or from fitted parameters renamed as predictions. The central 6.83-point gain is computed by comparing the adapted model with the unadapted Qwen-ASR-1.7B under the same shared diarization output (§4.1), which is a controlled experimental comparison rather than a circular reduction. The final evaluation-set tcpMER (17.97) is reported 'according to the task leaderboard' (§1, §5), providing an independent official check. The main caveat is that the development-set score (23.70) is produced by the authors' 'local scoring pipeline' (§1, §4.1), while the abstract describes it as 'on the official development set'; this is a measurement-validity and reporting concern, not circularity, because no parameter is fit to the metric and the claim remains externally testable. The ablations decompose training-stage contributions empirically; they do not assume the conclusion. No self-citation is load-bearing: references such as Qwen3-ASR, 3D-Speaker, CAMP++, and OmniVoice are external tools/models, not the authors' prior work. No uniqueness theorem, ansatz smuggled via citation, or renaming of a known result is present. Accordingly, the paper is self-contained as an empirical system description and receives a circularity score of 0.
Axiom & Free-Parameter Ledger
free parameters (13)
- VAD posterior decision threshold =
0.5
- VAD SNR correction terms =
-0.1/0.3
- Spectral clustering cosine threshold (mer_cos) =
0.8
- Spectral clustering p-value (pval) =
0.012
- Minimum cluster size =
4
- Same-speaker merge gap =
0.8 s (disabled by default in dev)
- Similarity-guided augmentation cosine threshold =
0.76–0.83
- LoRA rank / alpha =
r=64, alpha=128
- GRPO repetition penalty weight lambda_r =
0.10
- GRPO length-deviation penalty weight lambda_l =
0.10
- Hallucination trigger thresholds =
length ratio > 2.0; unique-token ratio < 0.5; 3-gram >= 2 repeats; penalty -1.0
- GRPO sampling hyperparameters =
K=2, temperature 0.7, top-k 20, top-p 0.95
- Training schedules (SFT/LoRA/RL) =
SFT 4 ep, lr 2e-5; LoRA 5 ep, lr 5e-5; RL lr 1e-6, KL beta 0.005
axioms (6)
- domain assumption tcpMER as defined by the challenge organizers (WER for space-separated scripts, CER for Japanese/Korean/Thai, then unweighted average across 21 conditions) is the accepted evaluation; the paper inherits it without independent validation.
- domain assumption The official development set (~4 h per language) is a representative proxy for the evaluation distribution, and model selection on it does not materially overstate generalization.
- domain assumption The released Qwen3-ASR-1.7B and quoted baselines are fixed public systems, and evaluating them in the local pipeline with the same shared diarization output isolates ASR-component differences.
- domain assumption OmniVoice synthetic speech is acoustically and linguistically realistic enough that models trained on it transfer to real conversational audio.
- domain assumption The reward R1 = 1 − WER/CER computed on normalized text is an unbiased training proxy for the final aggregated tcpMER metric.
- standard math The clipped PPO/GRPO surrogate objective with group-relative advantages converges to a useful policy for a speech-LLM action space.
invented entities (1)
-
none
no independent evidence
read the original abstract
This paper describes our self-designed system for Task 1 of the MLC-SLM 2026 Challenge for multilingual two-speaker conversational speech. The system combines a modular speaker diarization front end with a challenge-adapted Qwen3-ASR-1.7B recognizer. The diarization front end performs voice activity detection, subsegment generation, CAMPPlus speaker embedding extraction, two-speaker spectral clustering, and RTTM-based audio segmentation. The resulting speaker-attributed segments are grouped by language or region and decoded by the adapted ASR model. For ASR adaptation, we first perform supervised full fine-tuning on the official training data, then apply LoRA fine-tuning with synthetic speech generated by a three-pipeline TTS-based synthetic speech augmentation framework, and finally refine the model using GRPO reinforcement learning with rewards based on WER/CER and penalties for hallucination, repetition, and length deviation. On the official development set, the full system achieves an average tcpMER of 23.70, reducing the error rate by 6.83 absolute points relative to the released Qwen-ASR-1.7B performance. On the final evaluation set, the system achieves an average tcpMER of 17.97. Ablation results show that supervised fine-tuning provides the largest gain, while synthetic-speech LoRA adaptation and reinforcement learning further improve robustness.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction Transcribing two-speaker conversational recordings into speaker-attributed text is a demanding multilingual task: a single system must simultaneously decide who spoke when, segment the audio accordingly, and recognize what was said in each language. The difficulty is compounded by short turns, overlapping speech, frequent speaker switches, and...
2026
-
[2]
System framework and training strategy Figure 1 illustrates the overall framework of our system. The pipeline contains three functional parts: speaker diariza tion, au- dio segmentation and multilingual ASR, and text normalizat ion with evaluation. The ASR model is trained separately and the n integrated into the inference pipeline. 2.1. Stage-wise infere...
Pith/arXiv arXiv 2026
-
[3]
Dataset 3.1. Training data The MLC-SLM 2026 Task 1 training set provides two-speaker conversational speech recorded at 16 kHz in quiet indoor en- vironments, with oracle segmentation and speaker labels [ 1]. It covers 21 language/region conditions, built on the origi nal 11-language release (English ≈500 h, other languages ≈100 h each, totaling ≈1,500 h) ...
2026
-
[4]
Results We report tcpMER on the official development set under the challenge evaluation protocol
Experiments 4.1. Results We report tcpMER on the official development set under the challenge evaluation protocol. For ASR decoding we use greedy decoding at temperature 0 with a maximum output length of 256 tokens; diarization uses CAMPPlus embeddings and spectral clustering with the speaker count fixed to two, f ol- lowed by RTTM-driven segmentation. The ...
-
[5]
Conclusion We present an end-to-end speaker-diarization and ASR syste m for the MLC-SLM 2026 Task 1 challenge, built on Qwen3- ASR-1.7B and adapted through full supervised fine-tuning, synthetic-speech LoRA, and GRPO-based reinforcement lear n- ing. A three-pipeline TTS-based synthetic speech augmenta - tion framework based on OmniV oice contributes 234,33...
2026
-
[6]
CAM++: A fast and efficient network for speaker verification using context- aware masking,
H. Wang, S. Zheng, Y . Chen, L. Cheng, and Q. Chen, “CAM++: A fast and efficient network for speaker verification using context- aware masking,” in Proceedings of INTERSPEECH, 2023
2023
-
[7]
The Multilingual Conversational Speech Language Model Challenge 2026,
MLC-SLM 2026 Challenge Organizers, “The Multilingual Conversational Speech Language Model Challenge 2026,” https://www.nexdata.ai/competition/mlc-slm, 2026, accessed: 2026-07-06
2026
-
[8]
3D-Speaker- Toolkit: An open-source toolkit for multimodal speaker ver ifica- tion and diarization,
Y . Chen, S. Zheng, H. Wang, L. Cheng, T. Zhu, R. Huang, C. Deng, Q. Chen, S. Zhang, W. Wang, and X. Li, “3D-Speaker- Toolkit: An open-source toolkit for multimodal speaker ver ifica- tion and diarization,” arXiv preprint arXiv:2403.19971, 2024
Pith/arXiv arXiv 2024
-
[9]
X. Shi, X. Wang, Z. Guo, Y . Wang, P . Zhang, X. Zhang, Z. Guo, H. Hao, Y . Xi, B. Y ang, J. Xu, J. Zhou, and J. Lin, “Qwen3-ASR technical report,” arXiv preprint arXiv:2601.21337, 2026
Pith/arXiv arXiv 2026
-
[10]
LoRA: Low-rank adaptation of large lan- guage models,
E. J. Hu, Y . Shen, P . Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large lan- guage models,” in International Conference on Learning Repre- sentations (ICLR), 2022
2022
-
[11]
OmniV oice: Towards om- nilingual zero-shot text-to-speech with diffusion langua ge mod- els,
H. Zhu, L. Y e, W. Kang, Z. Y ao, L. Guo, F. Kuang, Z. Han, W. Zhuang, L. Lin, and D. Povey, “OmniV oice: Towards om- nilingual zero-shot text-to-speech with diffusion langua ge mod- els,” arXiv preprint arXiv:2604.00688, 2026
Pith/arXiv arXiv 2026
-
[12]
A. Y ang, B. Y ang, B. Hui, B. Zheng, B. Y u, C. Zhou, C. Li, C. Li, D. Liu, F. Huang et al., “Qwen2 technical report,” arXiv preprint arXiv:2407.10671, 2024
Pith/arXiv arXiv 2024
-
[13]
Wespeaker: A research and production oriented speaker embedding learning toolkit,
H. Wang, C. Liang, S. Wang, Z. Chen, B. Zhang, X. Xiang, Y . Deng, and Y . Qian, “Wespeaker: A research and production oriented speaker embedding learning toolkit,” in IEEE Interna- tional Conference on Acoustics, Speech and Signal Processi ng (ICASSP), 2023
2023
-
[14]
FunASR: A fundamental end-to- end speech recognition toolkit,
Z. Gao, Z. Li, J. Wang, H. Luo, X. Shi, M. Chen, Y . Li, L. Zuo, Z. Du, Z. Xiao, and S. Zhang, “FunASR: A fundamental end-to- end speech recognition toolkit,” arXiv preprint arXiv:2305.11013, 2023
Pith/arXiv arXiv 2023
-
[15]
End-to-end neural speaker diarization wi th permutation-free objectives,
Y . Fujita, N. Kanda, S. Horiguchi, Y . Xue, K. Nagamatsu, and S. Watanabe, “End-to-end neural speaker diarization wi th permutation-free objectives,” in Proceedings of INTERSPEECH , 2019
2019
-
[16]
Meeting recognition w ith continuous speech separation and transcription-supported diariza- tion,
T. von Neumann and K. Kinoshita, “Meeting recognition w ith continuous speech separation and transcription-supported diariza- tion,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024, arXiv:2309.16482
Pith/arXiv arXiv 2024
-
[17]
Qwen2-Audio technical re - port,
Y . Chu, J. Xu, Q. Y ang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv , J. He, J. Lin, C. Zhou, and J. Zhou, “Qwen2-Audio technical re - port,” arXiv preprint arXiv:2407.10759, 2024
Pith/arXiv arXiv 2024
-
[18]
MLC-SLM 2026 Task 1 Official Baseline,
MLC-SLM 2026 Challenge Organizers, “MLC-SLM 2026 Task 1 Official Baseline,” https://github.com/alanshaoTT/MLC-SLM-2nd-Task1-Baseline, 2026, accessed: 2026-07-06
2026
-
[19]
A. Y ang, A. Li, B. Y ang, B. Zhang, B. Hui, B. Zheng, B. Y uet al., “Qwen3 technical report,” arXiv preprint arXiv:2505.09388 , 2025
Pith/arXiv arXiv 2025
-
[20]
DeepSeekMath: Pushing the limits of mathematical reasoning in open language model s,
Z. Shao, P . Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . K. Li, Y . Wu, and D. Guo, “DeepSeekMath: Pushing the limits of mathematical reasoning in open language model s,” arXiv preprint arXiv:2402.03300, 2024
Pith/arXiv arXiv 2024
-
[21]
Text-to-speech data augmentation for lo w resource speech recognition,
R. Zevallos, “Text-to-speech data augmentation for lo w resource speech recognition,” arXiv preprint arXiv:2204.00291, 2022
Pith/arXiv arXiv 2022
-
[22]
Frustratingly easy data augm entation for low-resource ASR,
K. Ibaraki and D. Chiang, “Frustratingly easy data augm entation for low-resource ASR,” arXiv preprint arXiv:2509.15373, 2025
arXiv 2025
-
[23]
Com - mon V oice: A massively-multilingual speech corpus,
R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. H en- retty, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Com - mon V oice: A massively-multilingual speech corpus,” inProceed- ings of the Twelfth Language Resources and Evaluation Confe r- ence, 2020, pp. 4218–4222
2020
-
[25]
ASR er- ror correction using large language models,
R. Ma, M. Qian, M. J. F. Gales, and K. Knill, “ASR er- ror correction using large language models,” arXiv preprint arXiv:2409.09554, 2024
Pith/arXiv arXiv 2024
-
[512]
This stage serves as the final ASR refinement after supervised fine-tuning and LoRA adaptation
A low-variance KL loss regularizes the policy toward th e reference with β = 0 .005; KL is not injected into the reward. This stage serves as the final ASR refinement after supervised fine-tuning and LoRA adaptation
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.