{"id":"c500eb85-594c-4585-bc5e-14791b935852","arxiv_id":"2509.04392","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM-based ASR error correction framework with noise-adaptive encoding and dynamic multi-modal fusion reports WER gains, but its fusion weights require ground-truth text at inference.","lead":"This paper proposes an LLM-based speech error correction system that adds a noise-cleaning adapter to a frozen speech recognizer, fuses text and audio features with a dynamic weighting scheme, and trains with an RL objective on word error rate. The reported word error rates are lower than prior methods, but the dynamic fusion weights in the paper depend on the correct transcription, which is not available when the system runs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HFCDF fusion weights depend on ground-truth y (Eq. 11-12) with no inference-time substitute specified; the method as written cannot run on test data and reported gains may reflect label leakage.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: HFCDF uses the ground-truth target y to compute fusion weights, leaving the inference procedure unspecified. This is a fundamental correctness issue because the method, as written, cannot be executed on test data without leaking label information. The paper does not describe any inference-time substitute for y, and no code is provided to clarify. The Table IV contradiction further erodes confidence in the additive contributions of the components. Therefore, I agree with the reader's REJECT verdict; no additional independent concern is needed. My read does not change the verdict, so the recommendation is UNCHANGED.","tokens_in":12534,"tokens_out":2349,"duration_ms":23685,"concrete_test":"Obtain or request the official inference code for the full model. Locate the forward pass corresponding to Eq. 11. If Ra/Rt are computed using the ground-truth transcript y at inference, the method has label leakage and the reported WERs are invalid. If y is replaced by the 1-best hypothesis or a model-generated proxy, rerun the Table III comparison (HFCDF vs. Concat vs. Transformer) with that substitution and check whether the claimed improvements persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core fusion mechanism (Section III-D, Eqs. 11-12) computes dynamic weights as softmax over cosine similarities between each modality's features and the target output y. The paper states 'y represents the target output'—i.e., the ground-truth transcript. At inference, no reference transcript exists, and the paper provides no replacement (e.g., 1-best hypothesis, model prediction, or learned estimator). Consequently, the method as described is undefined at test time. This is not a minor implementation detail: the claimed superiority of HFCDF over Concat/Transformer (Table III) rests on these oracle-derived weights. If the authors used the ground-truth y during evaluation, the results are not achievable in practice. Additionally, Table IV shows the full model (5.32% WER on CHiME-4 test-real) is worse than the RL-loss-only ablation (4.92%), undermining the claim that all components together yield the best result. Both issues make the central claims unverifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Denoising GER, a multi-modal generative error correction framework for noisy ASR. It combines a noise-adaptive acoustic encoder (NAAE), a heterogeneous feature compensation dynamic fusion (HFCDF) mechanism, and a minimum-WER reinforcement learning loss (RL-loss) to improve LLM-based error correction. The authors report WER improvements on synthetic and real noisy datasets (DNS LibriSpeech, CHiME-4), unseen noise (MUSAN), and clean LibriSpeech, and present ablations supporting each component. However, the HFCDF mechanism computes dynamic fusion weights using the ground-truth target transcription y, with no inference-time substitute, so the method as described is not deployable and the reported gains may reflect label leakage. Additionally, Table IV shows the full model is worse than an ablation on CHiME-4 test-real, contradicting the claim that all components together give the best result.","tokens_in":12816,"tokens_out":2970,"duration_ms":28987,"significance":"If the method were sound, it would be a relevant contribution to LLM-based ASR post-processing, showing how to adapt acoustic encoders, fuse modalities, and optimize WER with RL. The paper has some commendable features: it uses public models (Whisper-Large v2, Qwen-7B), keeps trainable parameters small (26M), and includes experiments on in-domain, out-of-domain, and clean speech. However, the central HFCDF mechanism is oracle-dependent, which invalidates the main technical contribution and the empirical claims in Tables I and III. The inconsistency in Table IV further undermines the component-wise claims. The significance of the work cannot be assessed until the fusion mechanism is redefined and results are re-obtained without using the reference transcript.","major_comments":[{"comment":"The dynamic fusion weights μ are computed as softmax over cosine similarities between each modality feature (X′tok, Y′tok) and y, where y is defined as the target output (i.e., ground-truth transcription). At inference time, the ground-truth y is unavailable, and the paper supplies no replacement (e.g., the 1-best hypothesis, the ASR output, or a learned estimator). As written, the method cannot be run on test data; if the reference was used during evaluation, the reported WERs in Tables I and III are not achievable in practice and constitute label leakage. This is a load-bearing flaw, not a minor implementation detail, because Table III credits HFCDF's gain over Concat/Transformer fusion to these oracle-derived weights. The authors must specify an inference-time substitute and rerun all experiments; otherwise the central claims are unverifiable.","section":"§III-D, Eqs. (11)–(13)"},{"comment":"Table IV directly contradicts the claim that combining all components yields the best result. On CHiME-4 test-real, the full model (NAAE + HFCDF + RL-loss) gives WER 5.32, while RL-loss alone gives 4.92 and NAAE+HFCDF gives 4.75. The text states that adding RL-loss enhances performance and that the full model is optimal, but the table shows a clear degradation on a real noisy test set after adding RL-loss to NAAE+HFCDF. This inconsistency must be resolved: either the table is wrong, or the claim that the complete framework improves robustness is unsupported. The ablation conclusions in Section V-B need to be revised with a correct and consistent set of results.","section":"Table IV"},{"comment":"The notation in the NAAE formulation is ambiguous and inconsistent, which undermines reproducibility. Eq. (3) writes Xin = fθ(Xin) = X + adapterθ(Xin), but X is undefined and Xin appears on both sides with different roles. Eq. (5) uses L1(X′in, Xin) where X′in is introduced without definition, and λ is not specified in the experimental setup (only α and β are given). Since NAAE is one of the three main contributions, the objective being optimized must be stated unambiguously, with all variables defined and all hyperparameters specified.","section":"§III-B, Eqs. (3)–(5)"}],"minor_comments":[{"comment":"Typo: \"CHINE4\" should be \"CHiME-4\".","section":"§IV-A"},{"comment":"The contribution list says \"speech-text multi-modal featre HFCDF mechanism\" — 'featre' appears to be a typo for 'feature'.","section":"Abstract / §I Contributions"},{"comment":"The error analysis cites \"UADF [12] and MMGER [14]\" but the reference numbering does not match the bibliography entries; in the related work, MMGER is [16]. Please harmonize citations.","section":"§V-C"},{"comment":"The RL loss uses w(yi, y∗) but y∗ is not explicitly defined; it later appears as the ground-truth. Define y∗ consistently with y used in Eq. (11), and note whether the same ground-truth is used for the fusion weights during training.","section":"§III-E, Eq. (15)"},{"comment":"The figure contains typos: \"Languege\" and \"Embing\" should be \"Language\" and \"Embedding\". Also, the U-Net adapter structure is not clearly visible in the captioned subfigures; please improve the figure quality.","section":"Figure 1"},{"comment":"The hyperparameter k in Eq. (10) is introduced but its value is not reported in Section IV-B. Experimental reproducibility would benefit from listing k along with α and β.","section":"§III-D"}],"recommendation":"reject","confidential_remarks":"The oracle-dependence of the HFCDF fusion weights is a fundamental issue that cannot be fixed with a local correction; it changes the method and requires rerunning all experiments. The Table IV inconsistency further suggests the reported ablations may not be reliable. I would strongly encourage the authors to resubmit only after redefining the fusion mechanism in a deployable way and providing a consistent set of experiments, ideally with code for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clearly written engineering paper with a sensible three-part recipe for noise-robust GER, but the core fusion mechanism leaks the ground-truth transcript into the model at training and, as written, has no defined inference-time behavior. That is a load-bearing flaw, not a nitpick. It also has an ablation that undercuts its own 'everything helps' story.\n\nWhat's actually new: combining a U-Net adapter on a frozen Whisper encoder with difference-based cross-modal compensation and a minimum-WER RL loss is a reasonable assembly of known ideas. The dynamic n-best hypothesis generation by fine-tuning the ASR decoder is a nice touch. The experiments cover in-domain synthetic noise (DNS), real CHIME-4, unseen MUSAN, and clean speech, which is a sensible test protocol. The ablations in Tables II and III are well-structured and, in themselves, tell a coherent story about the value of fine-tuning the encoder over freezing or full FT, and about HFCDF over concat/transformer.\n\nThe problems: Eqs. 11–12 compute the fusion weight µ as a softmax over the cosine similarity between each modality embedding and the ground-truth target output y. No inference-time replacement (e.g., the 1-best hypothesis, a predicted transcript, or a learned estimator) is specified. So the model as described cannot be run on test speech; the reported 'inference results' are obtained with oracle fusion weights, which means the comparisons in Tables I and III are not achievable in practice. The paper needs to either replace y with an available estimate, or add a small network that predicts the weights, and then re-run all experiments with that choice.\n\nSecond, Table IV shows the full model (row 6) at 5.32 WER on CHIME4 test-real, while the RL-only model (row 4) gets 4.92 and NAAE+HFCDF (row 5) gets 4.75. The text claims adding RL-loss improves the framework, but on that dataset it makes things worse. That is a real inconsistency that the 'all components help' narrative ignores.\n\nMinor: no error bars, no code/data release, a number of typos. The related work is generally sound, with the usual self-citation caveats not standing out.\n\nVerdict: reject in current form. The label leakage is fatal to the empirical claims. But the underlying idea of making the acoustic encoder noise-adaptive and using dynamic fusion is worth pursuing; if the authors fix the fusion and re-evaluate, a revision could be considered. I would not cite this version.","headline":"Ground-truth transcript used to compute fusion weights with no inference-time recipe; Table IV also contradicts the 'all components help' narrative.","tokens_in":13296,"tokens_out":5054,"would_cite":false,"duration_ms":39946,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims Denoising GER, a three-part framework around a frozen LLM, lowers WER on noisy ASR post-processing and generalizes to unseen noise and clean speech.","keywords":["automatic speech recognition","generative error correction","large language models","multi-modal fusion","noise robustness","reinforcement learning","word error rate","acoustic adaptation"],"falsifier":"A concrete check: rerun Table I with HFCDF weights computed from the ASR top-1 hypothesis instead of the ground-truth y. If WER is unchanged, the target-dependence is benign; if WER rises, the HFCDF gains as reported depend on information that is not available in real inference.","tokens_in":12440,"feed_emoji":"🎙️","tokens_out":9307,"duration_ms":81651,"temperature":0.7,"pith_summary":"Generative error correction (GER) for speech recognition post-processes an ASR system's n-best hypotheses with a large language model to produce a better transcription. This paper argues that GER can be made noise-robust by adding three components to a frozen LLM: a noise-adaptive acoustic encoder that cleans the acoustic features before they reach a frozen Whisper encoder, a heterogeneous-feature compensation dynamic fusion (HFCDF) mechanism that reconciles speech and text embeddings before the LLM sees them, and a reinforcement-learning loss that directly minimizes word error rate. The authors show WER reductions on synthesized noisy LibriSpeech, real noisy CHIME4, unseen MUSAN noise, and clean test sets, with the full framework beating text-only GER and prior multi-modal GER baselines. If the result holds, it means noisy ASR can be corrected more accurately without retraining the large model and without sacrificing clean-speech performance.","feed_headline":"Denoising GER cuts WER on noisy speech and unseen noise","feed_subtitle":"A noise-adaptive acoustic front end, dynamic fusion, and WER-based RL make LLM error correction work in noise.","key_machinery":"The load-bearing machinery is the three-component training objective and the fusion identity inside HFCDF. NAAE is a U-Net adapter on the input spectrogram: X_in = X + adapter_θ(X_in), trained with a weighted cross-entropy plus L1 reconstruction loss, so the frozen ASR encoder sees noise-adapted features and yields dynamic n-best hypotheses. HFCDF uses the modality difference vectors Δx = X_tok − Y_tok and Δy = Y_tok − X_tok to form compensated features, then computes fusion weight µ as softmax over cosine similarities between each compensated feature and the target output; the final multimodal input is Concat(µX'_tok, (1−µ)Y'_tok−top1). RL-loss is the minimum-WER policy-gradient term L_RL =","core_discovery":"The paper's central claim is that Denoising GER—the combination of NAAE, HFCDF, and RL-loss—achieves the best WER on noisy speech while retaining accuracy on clean speech. NAAE inserts a small trainable U-Net adapter before a frozen Whisper-Large v2 encoder; the adapter transforms noisy Mel spectrograms back toward a clean-like representation and lets the ASR decoder emit a dynamic n-best list instead of a fixed one. HFCDF then computes difference vectors between acoustic and text embeddings, applies cross-modal compensation, and weights the two compensated streams by softmax cosine similarity to the target. The LLM (Qwen-7B, LoRA-tuned) receives the fused frame-level multimodal representati","pith_inferences":["Because the HFCDF fusion weight is defined through the ground-truth transcription y, a test-time implementation needs a substitute reference (for example, the ASR 1-best hypothesis); the paper does not specify one, and checking whether the gains survive this substitution is a direct way to test the mechanism.","The NAAE input-reconstruction idea is a general learned denoising front end; it could be applied to other ASR post-processing tasks or to streaming ASR if the U-Net adapter is made causal.","The same difference-vector compensation plus similarity weighting could be transferred to other paired modalities, such as audio-visual speech, where one modality is noisier than the other."],"forward_implications":["Noisy ASR post-processing can be improved by adapting the acoustic front end; the full framework needs only 26M trainable parameters around a frozen 7B LLM.","Dynamic, similarity-weighted fusion of acoustic and text streams can outperform fixed concatenation because it reduces the speech-text modality gap.","Adding a minimum-WER reinforcement objective to cross-entropy improves sentence-level correction in noise without hurting character-level training.","The noise adaptation transfers: gains on in-domain noisy data persist on unseen MUSAN noise and on clean LibriSpeech, so robustness does not trade away clean accuracy."],"supporting_citations":[{"why":"Defines the GER benchmark and the text-only Hyporadise baseline that Denoising GER extends and must outperform.","marker":"[10]"},{"why":"Represents the noise-robust GER baseline using knowledge distillation, compared in Tables I and V.","marker":"[11]"},{"why":"MMGER provides the multi-modal GER baseline and the prompt-concatenation approach that HFCDF is designed to improve.","marker":"[16]"},{"why":"Qwen-Audio is a multi-modal audio-LLM baseline compared in Table I and a reference for integrating Whisper-based acoustic features.","marker":"[17]"},{"why":"Whisper-Large v2 is the frozen ASR encoder whose features NAAE adapts and whose decoder generates the n-best hypotheses.","marker":"[31]"},{"why":"LibriSpeech supplies the train-clean-100 training set and the test-clean/test-other clean and synthetic noisy evaluation sets.","marker":"[34]"},{"why":"Supplies the interpolated cross-entropy plus RL minimum-WER training idea that the RL-loss is based on.","marker":"[37]"},{"why":"MUSAN provides the out-of-domain noise corpus used to test generalization to unseen noise.","marker":"[38]"},{"why":"CHIME4 supplies the real-recorded distant-microphone noisy speech used as an in-domain real-noise test.","marker":"[40]"},{"why":"Qwen-7B is the frozen LLM that carries out the generative error correction.","marker":"[41]"}],"fun_headline_variants":["Denoising GER: LLM error correction that resists noise","Noise-adaptive fusion cuts WER on unseen noise","RL-trained LLM cleans up noisy ASR output","Multimodal GER tackles noisy speech and unseen noise"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The HFCDF fusion weights are defined using the ground-truth transcription y, and the paper does not state what value is used at inference when y is unknown; the reported test-time gains therefore rest on an unspecified stand-in for the target.","fun_headline_variants_meta":{"raw":{"variants":["Denoising GER: LLM error correction that resists noise","Noise-adaptive fusion cuts WER on unseen noise","RL-trained LLM cleans up noisy ASR output","Multimodal GER tackles noisy speech and unseen noise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1297,"prompt_tokens":705,"completion_tokens":592,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":525}},"tokens_in":449,"tokens_out":592,"duration_ms":5814,"temperature":1.0,"reasoning_tokens":525,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:12:24.407583+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: rerun Table I with HFCDF weights computed from the ASR top-1 hypothesis instead of the ground-truth y. If WER is unchanged, the target-dependence is benign; if WER rises, the HFCDF gains as reported depend on information that is not available in real inference.","supporting_citations":[{"cited_title":"Hyporadise: An open baseline for generative speech recognition with large language models,","cited_arxiv_id":null,"evidence_quote":"Defines the GER benchmark and the text-only Hyporadise baseline that Denoising GER extends and must outperform."},{"cited_title":"Noise-robust speech recognition with 10 minutes unparalleled in-domain data,","cited_arxiv_id":null,"evidence_quote":"Represents the noise-robust GER baseline using knowledge distillation, compared in Tables I and V."},{"cited_title":"Mmger: Multi-modal and multi-granularity generative error correction with llm for joint accent and speech recognition,","cited_arxiv_id":null,"evidence_quote":"MMGER provides the multi-modal GER baseline and the prompt-concatenation approach that HFCDF is designed to improve."},{"cited_title":"Robust speech recognition via large-scale weak supervi- sion,","cited_arxiv_id":null,"evidence_quote":"Whisper-Large v2 is the frozen ASR encoder whose features NAAE adapts and whose decoder generates the n-best hypotheses."},{"cited_title":"Librispeech: an asr corpus based on public domain audio books,","cited_arxiv_id":null,"evidence_quote":"LibriSpeech supplies the train-clean-100 training set and the test-clean/test-other clean and synthetic noisy evaluation sets."},{"cited_title":"The rwth/upb/forth system combination for the 4th chime challenge evaluation,","cited_arxiv_id":null,"evidence_quote":"CHIME4 supplies the real-recorded distant-microphone noisy speech used as an in-domain real-noise test."}],"review_version":1}