{"id":"30b6eba3-02af-4341-888e-b6f56cb9e84c","arxiv_id":"2506.13596","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Whisper-large-v3 encoder with a linear projector and LoRA-tuned Gemma3-12B decoder achieves 16.63% average WER/CER on the MLC-SLM 2025 private test set.","lead":"This paper describes a system for the 2025 MLC-SLM multilingual speech challenge, combining a fine-tuned Whisper encoder with either Qwen2.5-7B or Gemma3-12B as a language model decoder. On the private test set, the best configuration reaches an average word/character error rate of 16.63%.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Qwen-vs-Gemma conclusion varies decoder family, model size, projector compression, LoRA precision, and stage-2 encoder training at once; Section 4.1's 'significantly' claim is not yet supported.","rationale":"The paper is best read as a challenge-system report: its main deliverable is an absolute private-test WER/CER of 16.63% with a Gemma3-12B decoder. That number is plausible, consistent with the development-set ordering, and supported by a linked repository and a stated leaderboard rank, so I do not dispute it. The paper, however, frames itself as a comparative study ('Qwen vs. Gemma') and explicitly concludes that Gemma3 'helps to reduce the CER/WER significantly.' That inference is the most load-bearing part of the argument because it is what would generalize beyond this specific submission. The configuration sections vary model family, model size, projector compression, LoRA precision, and the stage-2 encoder-update rule simultaneously, so the 1.97-point gap on the private test cannot be cleanly attributed to the decoder choice. The development-set gap is only 0.63 points, and the absence of uncertainty estimates means even that comparison is fragile. The reader's weakest-assumption analysis already identified the confounding of projector and precision differences; I agree, and I would add decoder scale as an additional confound. Because the existing CONDITIONAL verdict already requires controlled experiments and error bars, my stress-test does not change the recommended verdict; it sharpens the specific experiment that would settle the point.","tokens_in":6076,"tokens_out":6564,"duration_ms":71228,"concrete_test":"Using the released repository, run a 2x2 controlled comparison on the development set: Qwen2.5-7B and Gemma3-12B, each with Projector 4 and Projector 5, with matched LoRA precision and matched stage-2 encoder training, plus at least three seeds and per-utterance bootstrap 95% confidence intervals. If the Gemma advantage falls within the confidence intervals or flips when the projector or precision is matched, the Section 4.1 attribution is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparative claim is that Gemma3 reduces WER/CER significantly compared with Qwen2.5-7B. But Sections 3.3.2 and 3.3.3 change five variables simultaneously: decoder family and size (Gemma3-12B vs. Qwen2.5-7B), projector compression (Projector 4 vs. Projector 5), LoRA precision (4-bit vs. 16-bit), and stage-2 encoder training (explicitly stated only for Gemma3). Section 4.1 then attributes the 1.97-point private-test gap to 'the integration of Gemma3.' On the development set, the same comparison is only 0.63 points (20.68 vs. 21.31), and no confidence intervals, per-utterance variance, or repeated-seed results are reported. Any one of the confounds, or sampling noise, could explain the observed gap. The absolute system performance of 16.63% on the private test is not the problem; the unsupported inference is the claim that Gemma3 as a decoder family is responsible for the improvement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports a modular multilingual SpeechLLM system submitted to the MLC-SLM Challenge 2025. It couples a fine-tuned Whisper-large-v3 encoder with a two-layer SwiGLU linear projector and a LoRA-tuned decoder LLM (either Qwen2.5-7B or Gemma3-12B), using a three-stage training schedule. On the challenge's private test set, it reports an average WER/CER of 16.63% with Gemma3-12B and 18.60% with Qwen2.5-7B, corresponding to rank 15/25, and it also reports per-language development-set results and ablations against Whisper-only, Qwen, Llama, and Phi-4 baselines. The main textual claim is that integrating Gemma3 significantly reduces WER/CER compared with Qwen2.5-7B.","tokens_in":6307,"tokens_out":6313,"duration_ms":55978,"significance":"The absolute result is a useful datapoint: a relatively simple, modular pipeline (fine-tuned encoder + linear projector + LoRA LLM) reaches competitive multilingual conversational ASR performance, and the paper includes per-language development-set tables, an error-correction ablation, and a public code repository. These are real strengths. The comparative significance of the Gemma3-vs-Qwen result is not yet established, because the two configurations differ in several design variables and no uncertainty estimates are given. As a system/competition report, the paper is worth publishing after the comparative claim is either tested in a controlled way or substantially weakened.","major_comments":[{"comment":"The claim that 'The integration of Gemma3 helps to reduce the CER/WER significantly' is not supported by the current experimental design. The Qwen2.5-7B and Gemma3-12B systems differ simultaneously in decoder family and size, projector compression ratio (Projector 5 vs Projector 4, §3.1.2), LoRA precision (16-bit vs 4-bit), and in whether stage 2 continues to train the speech encoder (explicitly stated only for Gemma3 in §3.3.3). Any of these factors, or their interaction, could account for the 1.97-point private-test gap. Please add controlled configurations that vary only the decoder (ideally also reporting matched projector and LoRA settings) or, if that is not feasible within the competition setting, revise the conclusion so that the improvement is attributed to the Gemma3-based configuration rather than to Gemma3 as a decoder family.","section":"§3.3.2, §3.3.3, §4.1"},{"comment":"All results appear to come from a single run, with no confidence intervals, standard deviations, or repeated-seed experiments. The development-set gap between the two systems is only 0.63 points (21.31 vs 20.68), while the private-test gap is 1.97 points (18.60 vs 16.63); without variance information, the word 'significantly' in §4.1 is not justified statistically. Please report at least a small number of independent runs or an equivalent uncertainty estimate for the central comparison, and otherwise use a non-statistical phrasing such as 'in our experiments, the Gemma3-based configuration achieved a lower WER/CER.'","section":"§4.1, Table 1"},{"comment":"The 'Baseline' row (20.17) is used to compute relative improvements of 7.78% and 17.55%, but the manuscript never states what this baseline is (official challenge baseline, fine-tuned Whisper, or some other model). Without this definition, the relative gains in §4.1 cannot be interpreted. Please identify the baseline system and confirm that the comparison is on the same development/private test conditions.","section":"§4.1, Table 1"}],"minor_comments":[{"comment":"Please state explicitly, for each configuration, which components are frozen vs trainable in each of the three stages; the current text implies Qwen and Gemma differ in stage 2 but does not spell out the Qwen stage-2 setup.","section":"§3.3"},{"comment":"The columns 'Baseline LargeV3' and 'LargeV3-I' both appear to refer to fine-tuned Whisper-large-v3; rename them so the reader can tell organizer baselines from the authors' reproductions.","section":"Table 2"},{"comment":"Reference [3] is cited for Llama3.1-8B but the reference points to Llama-Adapter v2; add the correct citation for Llama 3.1.","section":"References"},{"comment":"The sentence about Gemma3-12B-4bit-III vs Baseline-Qwen says the model 'slightly underperforms Baseline-Qwen with a marginal increase of 0.29%,' but Table 2 shows Gemma3-12B-4bit-III at 20.68% vs Baseline-Qwen at 21.31%, i.e., Gemma is better; the stated numbers appear inconsistent.","section":"§4.2"},{"comment":"Table 2 contains implausible entries for Baseline-Qwen on Russian (608.22) and Korean (420.00); if these are formatting artifacts or special cases, please explain them or correct them.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The review is based on the submitted arXiv v2. The core system result is likely sound as an engineering claim, but the title and §4.1 overstate the decoder comparison. If the authors reframe the paper as a system description with the decoder comparison as an observed trend rather than a causal conclusion, I would be inclined to accept after minor revision; with the current framing, major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent challenge entry. The system is a modular Whisper-large-v3 + linear projector + LoRA-tuned LLM, trained in three stages, and it reaches 16.63% average WER/CER on the MLC-SLM 2025 private test (rank 15/25), beating the organizer baseline by a comfortable margin. Code is linked. That part is real and useful for practitioners.\n\nThe weak spot is Section 4.1's conclusion. Sections 3.3.2 and 3.3.3 change five variables at once when comparing Qwen2.5-7B and Gemma3-12B: decoder family and size, projector compression ratio (5:1 vs 4:1), LoRA precision (16-bit vs 4-bit), and whether stage 2 trains the encoder. Private-test gap is 1.97 points (18.60 vs 16.63), but the development-set gap is only 0.63 (21.31 vs 20.68), with no variance estimates or repeated seeds. So \"the integration of Gemma3 helps to reduce the CER/WER significantly\" is not supported by the evidence presented. The stress-test note is correct on this point.\n\nAlso minor: Table 2 contains an anomalous value (Baseline-Llama Russian at 608.22) that looks like a typo and should be fixed. The paper's overall framing is humble — it is a system description, not a claim to a new paradigm — so the confound is the main substantive issue.\n\nWhat the paper does well: the three-stage schedule is clearly described; the comparison against several baselines (vanilla Whisper, Baseline-Qwen, Baseline-Llama, Phi-4 zero-shot) gives useful context; and the result is placed within a formal challenge setting with a defined test set. The code release helps reproducibility.\n\nWho should read it: engineers building multilingual speech-LLMs, and people looking for challenge baselines. It is not going to change any research direction, but it is a legitimate data point.\n\nI would not desk-reject it. A serious referee can ask for controlled experiments (matched projector, precision, and training schedule) and error bars. The absolute result and the system description justify referee time. My recommendation: send to peer review, and require the comparative conclusion to be softened or backed by those controlled runs.","headline":"The absolute result is solid, but the Gemma-vs-Qwen comparison is confounded and the 'significantly' claim needs controlled experiments before it can be taken seriously.","tokens_in":6812,"tokens_out":2663,"would_cite":false,"duration_ms":26592,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a Whisper-large-v3 encoder linked to a Gemma3-12B decoder through a lightweight projector reaches 16.63% average WER/CER on the MLC-SLM 2025 private test, beating the Qwen2.5-7B configuration by 1.97 points.","keywords":["multilingual speech recognition","speech language models","Whisper","Qwen2.5","Gemma3","LoRA","speech encoder projector","WER/CER"],"falsifier":"Rerun the experiment with the projector compression ratio, LoRA precision, and stage-2 encoder training held identical across Qwen2.5-7B and Gemma3-12B; if the 1.97-point gap shrinks or reverses, the paper's attribution of the improvement to the decoder choice is not supported.","tokens_in":5896,"feed_emoji":"🎙️","tokens_out":6679,"duration_ms":62606,"temperature":0.7,"pith_summary":"The paper argues that a modular speech-language model built from a fine-tuned Whisper-large-v3 encoder, a lightweight two-layer SwiGLU projector, and a LoRA-adapted decoder-only LLM can reach competitive multilingual speech recognition on the MLC-SLM Challenge 2025 test set. Its central comparative claim is that swapping the decoder from Qwen2.5-7B to Gemma3-12B lowers the average WER/CER from 18.60% to 16.63% on the private test set. The three-stage training schedule, which optimizes the encoder, then the encoder with the projector, then the projector with the LoRA adapter, is presented as the method that makes this integration work. A sympathetic reader would take the paper as evidence that a simple pretrained-component stack can perform well without heavily customized engineering.","feed_headline":"Whisper plus Gemma3 reaches 16.63% on multilingual speech","feed_subtitle":"A fine-tuned Whisper encoder, a small projector, and a LoRA-tuned Gemma3 decoder beat the Qwen2.5 variant by 1.97 points.","key_machinery":"The central mechanism is the three-stage training pipeline that connects a Whisper-large-v3 speech encoder, a two-layer SwiGLU perceptron projector, and an instruction-tuned decoder LLM adapted with LoRA. The projector performs temporal compression, reducing the 1,500 frames of a 30-second segment to 300 frames (5:1 compression) for Qwen2.5-7B or 375 frames (4:1 compression) for Gemma3-12B before the representation enters the LLM. Stage 1 trains only the encoder; stage 2 trains the encoder with the projector; stage 3 trains the projector together with the LoRA adapter while the rest of the LLM stays frozen. This staged alignment of speech features with the LLM's text space is the load-bearing mechanism that the paper claims makes the system work.","core_discovery":"The paper reports that its Gemma3-12B configuration outperforms its Qwen2.5-7B configuration on the MLC-SLM Challenge 2025 development and private evaluation sets, with a private-test average WER/CER of 16.63% versus 18.60%. The authors attribute this 1.97-point absolute gap to the choice of language model, stating that the integration of Gemma3 helps to reduce the CER/WER significantly. The same experiments show that a cascaded setup in which the LLM corrects Whisper's output raises the error from 17.67% to 31.29% on the development set, while the end-to-end three-stage system keeps it lower, which the paper takes as evidence that joint optimization of the encoder, projector, and LLM is what makes the integration effective. In the challenge ranking, the Gemma3 system placed 15 out of 25.","pith_inferences":["The paper's own setup cannot isolate the decoder's contribution: the Qwen and Gemma runs differ in projector compression, LoRA precision, and whether the encoder is trained in stage 2, so the 'Gemma helps' claim is a hypothesis rather than a controlled result.","A natural testable extension is to rerun both decoders with identical projector, precision, and stage schedule, and with multiple random seeds, to see whether the 1.97-point gap survives.","Although the paper does not draw this conclusion, if the modular recipe generalizes, future systems could mix and match speech encoders and LLMs as drop-in parts, decoupling speech-model progress from LLM progress."],"forward_implications":["A pretrained Whisper encoder can serve as a multilingual speech front-end for instruction-tuned LLMs, so a speech-LLM can be assembled from existing components rather than trained from scratch.","The 1.97-point gap suggests that decoder choice can matter at the level of a few error-rate points within a fixed encoder-projector pipeline, at least on this dataset.","End-to-end joint training beats a cascaded Whisper-plus-error-correction setup by a large margin, implying that post-hoc correction is not a substitute for aligned training.","The private-test average of 16.63% places the system 15th among 25 submissions, showing the approach is competitive in a shared multilingual benchmark."],"supporting_citations":[{"why":"Supplies the Whisper-large-v3 speech encoder that the system fine-tunes and uses as its speech front-end.","marker":"[7]"},{"why":"Defines the Qwen2.5-7B decoder LLM for one of the two compared configurations.","marker":"[8]"},{"why":"Defines the Gemma3-12B decoder LLM for the better-performing configuration.","marker":"[9]"},{"why":"Provides the encoder-projector-LLM integration approach that the paper adapts to Whisper and conversational speech.","marker":"[4]"},{"why":"Provides the Llama3.1-8B baseline that the paper compares against in its ablation study.","marker":"[3]"},{"why":"Supplies the Phi-4 multimodal LLM used as a zero-shot baseline in the language-by-language results.","marker":"[14]"},{"why":"Defines the SwiGLU activation used in the projector layers.","marker":"[10]"},{"why":"Supplies the SpecAugment augmentation applied to the speech encoder during training.","marker":"[13]"}],"fun_headline_variants":["Gemma3 beats Qwen2.5 by 2 points in multilingual speech","Qwen vs Gemma: Whisper integration, Gemma wins on WER/CER","Multilingual speech: Whisper+Gemma3 hits 16.63%, beats Qwen by 2 points","Whisper-Gemma3 wins multilingual challenge, Qwen2.5 lags","Gemma3 with Whisper: 16.63% WER/CER, 2-point edge over Qwen"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the better result comes from choosing Gemma3, but the Qwen and Gemma systems differ in three other ways—how much the speech features are compressed, how precisely the model adapter is tuned, and whether the speech encoder is trained in the middle stage—so the improvement cannot be pinned to the language model alone.","fun_headline_variants_meta":{"raw":{"variants":["Gemma3 beats Qwen2.5 by 2 points in multilingual speech","Qwen vs Gemma: Whisper integration, Gemma wins on WER/CER","Multilingual speech: Whisper+Gemma3 hits 16.63%, beats Qwen by 2 points","Whisper-Gemma3 wins multilingual challenge, Qwen2.5 lags","Gemma3 with Whisper: 16.63% WER/CER, 2-point edge over Qwen"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000621,"raw_usage":{"total_tokens":2834,"prompt_tokens":855,"completion_tokens":1979,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":1854}},"tokens_in":471,"tokens_out":1979,"duration_ms":11670,"temperature":1.0,"reasoning_tokens":1854,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:29:15.192606+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the experiment with the projector compression ratio, LoRA precision, and stage-2 encoder training held identical across Qwen2.5-7B and Gemma3-12B; if the 1.97-point gap shrinks or reverses, the paper's attribution of the improvement to the decoder choice is not supported.","supporting_citations":[{"cited_title":"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the Whisper-large-v3 speech encoder that the system fine-tunes and uses as its speech front-end."},{"cited_title":"Main results The main results are illustrated in Table 1","cited_arxiv_id":null,"evidence_quote":"Provides the encoder-projector-LLM integration approach that the paper adapts to Whisper and conversational speech."},{"cited_title":"Models 3.1.1","cited_arxiv_id":null,"evidence_quote":"Provides the Llama3.1-8B baseline that the paper compares against in its ablation study."}],"review_version":1}