{"id":"425232b1-c595-49d1-8182-73d90010e962","arxiv_id":"2412.03075","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The ASR-EC benchmark on Chinese ASR errors shows that multimodal LLM augmentation corrects ASR output best, while prompting alone worsens CER.","lead":"The paper creates a Chinese ASR error correction benchmark (ASR-EC) using two Kaldi ASR systems and evaluates LLMs with prompting, finetuning, and multimodal augmentation. It reports that multimodal audio-plus-text correction achieves the lowest character error rates, while prompting alone often increases errors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's 'Multimodal' rows are attributed to Baichuan2, ChatGLM3, and Qwen, which Table 3 lists as text-only models, even though Section 6 requires raw audio input; the central SOTA claim rests on these unexplained rows.","rationale":"I read the paper in good faith and agree that its central claim is the multimodal-augmentation result. The reader's weakest assumption focused on benchmark representativeness, specifically whether the Kaldi pipelines and the 10% retention of error-free utterances produce a representative error distribution. That is a real concern, but I see a more immediate threat to the central claim: the 'Multimodal' rows in Table 4 are attributed to three text-only LLMs with no described mechanism for ingesting audio. This is not a disagreement with an outside consensus; it is an internal inconsistency between Section 6, Table 3, and Table 4. If the multimodal rows are not reproducible as stated, the paper's headline conclusion has no valid supporting experiment. Table 5 provides useful evidence that finetuning Qwen-Audio with ASR-EC improves over vanilla Qwen-Audio ASR, but it does not compare audio-plus-text against text-only finetuning of the same model, so it cannot by itself establish that multimodal augmentation is the most effective paradigm. I would therefore move from CONDITIONAL to UNVERDICTED: the benchmark may be useful and some finetuning results may be valid, but the central comparative finding cannot be assessed until the model-input mismatch is resolved and the artifacts are released.","tokens_in":11711,"tokens_out":4920,"duration_ms":49327,"concrete_test":"Obtain the exact checkpoints and inference scripts that produced Table 4's 'Multimodal' rows and reproduce one cell, e.g., Baichuan2 on ASR-EC A* mixed utterances with CER 5.96, using the described audio-plus-text pipeline. If the released Baichuan2 checkpoint cannot accept audio input, or if the reproduction shows that the row was generated from text alone or with an undeclared audio encoder, the row is not a valid multimodal result and the central claim is unverified. As a second check, ablate the audio channel by feeding the same configuration with silence in place of the audio; if the CER is unchanged, the reported gain is not attributable to audio.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The paper's headline conclusion is that multimodal augmentation is the most effective approach and achieves state-of-the-art performance. The quantitative support for this is Table 4, which reports 'Multimodal' CERs of 5.96 for Baichuan2, 6.07 for Qwen, and 11.51 for ChatGLM3 on ASR-EC A* mixed utterances, all below the corresponding LoRA-finetuned CERs. However, Section 6 defines the multimodal input as the 'encoded raw audio input' concatenated with the error-correction prompt and the erroneous ASR text, and Table 3 states that Baichuan2, ChatGLM3, and Qwen are text-only models with no audio modality. No audio encoder, adapter, or input projection is introduced anywhere in Sections 6 or 7 for these three models. A text-only checkpoint cannot ingest raw audio, so either those rows were produced by some undeclared audio-capable pipeline, in which case the comparison is not the claimed controlled study of three paradigms, or they are text-only results mislabeled as multimodal, in which case the central SOTA conclusion is unsupported. Table 5 does not rescue the claim: it shows that Qwen-Audio finetuned with ASR-EC improves over the Qwen-Audio ASR mode, but it does not compare against text-only LoRA finetuning of the same base model, so it does not establish that adding audio is the decisive factor. This internal inconsistency is more load-bearing than the benchmark-representativeness concern because it undermines the paper's central comparative finding directly, before the evaluation distribution is even considered.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ASR-EC, a Chinese ASR error correction benchmark constructed by decoding audio from THCHS-30, AISHELL-1, AISHELL-2, and WeNetSpeech with two Kaldi-based pipelines (Kaldi-K1 and Kaldi-K2), splitting the resulting transcripts into short and long utterance subsets, and retaining only 10% of error-free utterances. It then evaluates three paradigms for applying LLMs to ASR error correction: prompting (zero-shot, three-shot, and multi-step), LoRA fine-tuning, and a multimodal augmentation approach that concatenates encoded raw audio with the instruction and the erroneous ASR text as input to the LLM. The reported findings are that prompting degrades CER relative to the ASR baselines, LoRA fine-tuning improves a subset of models, and multimodal augmentation achieves the lowest CERs, with an additional experiment showing that fine-tuning Qwen-Audio on ASR-EC improves over Qwen-Audio's ASR mode. The paper claims to present the first Chinese ASR error correction benchmark and the first study of LLM-based ASR error correction for Chinese.","tokens_in":12042,"tokens_out":5155,"duration_ms":47778,"significance":"If the benchmark construction is sound and the multimodal results are reproducible, ASR-EC would be a useful resource for the community: it aggregates four widely used Chinese speech corpora, provides error breakdowns by substitution/deletion/insertion, and covers both hybrid and end-to-end ASR decoding pipelines. The paper also provides a systematic comparison of prompting, fine-tuning, and multimodal input strategies, which is valuable for practitioners choosing among these paradigms. However, the central comparative claim that multimodal augmentation is the most effective approach is not currently supported because the multimodal rows in Table 4 are not tied to a concrete audio-capable model for Baichuan2, ChatGLM3, and Qwen, and the Qwen-Audio experiment in Section 7.4 does not isolate the contribution of the audio modality. The benchmark's representativeness is also asserted rather than demonstrated, so the generality of the conclusions remains uncertain.","major_comments":[{"comment":"The central claim that 'multi-modal augmentation ... achieves state-of-the-art performance' rests on rows in Table 4 that are internally inconsistent with the model descriptions in Table 3 and Section 6. Baichuan2, ChatGLM3, and Qwen are listed as text-only models in Table 3, yet Table 4 reports 'Multimodal' CER values for these models (e.g., 5.96 for Baichuan2 on ASR-EC A* mixed utterances). Section 6 defines the multimodal input as encoded raw audio concatenated with the instruction and erroneous ASR text, but no audio encoder, adapter, or input projection is described for these three checkpoints. The authors must either specify the audio-capable models or pipeline used to produce those rows, including any additional parameters or training steps, or remove/relabel them. As written, the comparative conclusion that multimodal augmentation is the most effective paradigm is unsupported.","section":"§7.2, Table 4 and §6"},{"comment":"The Qwen-Audio experiment does not test whether adding audio is the decisive factor. Table 5 compares Qwen-Audio in its ASR mode with Qwen-Audio fine-tuned on ASR-EC; both variants use audio and text inputs. To support the claim that audio plus transcript is more effective than transcript alone, the paper needs a controlled comparison with a text-only LoRA fine-tuned model (e.g., Qwen or Qwen-Audio with audio inputs ablated) trained on the same ASR-EC data. Without such a comparison, the observed improvement may simply reflect fine-tuning on the training distribution rather than the multimodal input format.","section":"§7.4, Table 5"},{"comment":"The benchmark construction uses an arbitrary 10% retention rate for utterances with CER 0 ('To control the number of those with a CER of 0, only 10% of them are kept'), but no sensitivity analysis or justification is given for this rate. Changing this rate can substantially alter the error distribution and the difficulty of the correction task, which directly affects all downstream comparisons. Additionally, the claim that Kaldi-K1 and Kaldi-K2 are 'the only two pipelines in the history of ASR systems' is unsupported and implausible, and the assertion that their error patterns are representative of industry-grade Chinese ASR systems is not backed by evidence. The authors should provide a justification for the 10% cap and temper or support the representativeness claim.","section":"§3.2"},{"comment":"All CER values are reported as point estimates without confidence intervals, significance tests, or multiple-run variability. Several comparisons that drive the conclusions involve small absolute differences (e.g., Baichuan2 vs Qwen multimodal on ASR-EC A* long utterances: 6.99 vs 5.64), and the ranking of models and paradigms may change within noise. The authors should report error bars or paired significance tests, or at least state the number of evaluation runs and the variance observed.","section":"Tables 4 and 5"},{"comment":"The paper claims to present 'the first Chinese ASR error correction benchmark' and 'the first one to study LLM-based ASR error correction,' but Section 2 itself cites earlier LLM-based ASR error correction work (references [10]–[15]) and the related-work discussion indicates substantial prior art. The novelty claims need to be qualified and reconciled with these references; as stated, they overstate the contribution.","section":"Abstract, Section 2, Section 3.1"}],"minor_comments":[{"comment":"There is a typo in the Substitution CER for ASR-EC B* short utterances: '9,59' should be '9.59'. In addition, the relationship between the test-set CER in the upper part of the table and the component CERs in the lower part is not explained (e.g., ASR-EC A* Whole Dataset has Overall CER 13.32 while the reported test CER is 12.42); the authors should clarify which split the component CERs are computed on.","section":"Table 2"},{"comment":"The sentence 'Prompting is an emerging technique for fine-tuning large language models' is inaccurate in standard terminology: prompting does not modify model parameters. Please rephrase to describe prompting as an inference-time adaptation method.","section":"Section 4"},{"comment":"The text refers to a 'one-shot' setting, but Table 4 reports 'Three-Shot.' Please make the terminology consistent and describe the exact number and selection of few-shot examples.","section":"Section 7.2"},{"comment":"Reference [2] is cited as 'Kaldi-K1' but the actual reference is the Kaldi toolkit paper; the paper should cite or specify the specific Kaldi-K1 recipe (e.g., the multistream CNN configuration) and the Kaldi-K2/Zipformer-Transducer source separately.","section":"Section 7.1 and References"},{"comment":"There is a typo in the contributions list: 'funetuning' should be 'finetuning.' Also, the terms 'fine-tuning' and 'funetuning' are used inconsistently throughout the paper.","section":"Contributions and Section 5"},{"comment":"Reference [1] points to an anonymous GitHub repository path containing 'train_data' only, while the text states the dataset will be released upon publication. The availability status of the full benchmark (including test sets and evaluation scripts) should be clarified.","section":"Reference [1]"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the most serious issue is the unexplained 'Multimodal' rows for text-only models in Table 4; this is not a stylistic problem but a gap in the main experimental claim. The benchmark release also appears to be behind an anonymous repository, so the resource's availability cannot currently be verified. If the authors can clarify the multimodal experimental setup and add controlled comparisons, the paper could become a solid empirical contribution; I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the good news: ASR-EC is a genuinely useful artifact if the \"first Chinese benchmark\" claim holds, which nothing here contradicts. The construction from four public Mandarin corpora via two different Kaldi pipelines (hybrid and end-to-end) is sensible, and the train/test splits and error-type breakdown are a solid basis for future work. The LoRA finetuning results are credible and show clear improvements over prompting, which is a useful negative result.\n\nThe problem is the multimodal section. The stress-test note is right: Table 4 reports \"Multimodal\" rows for Baichuan2, ChatGLM3, and Qwen, but Table 3 lists all three as text-only, and Section 6 says the method concatenates encoded raw audio. No audio encoder or adapter is introduced for these models. Either those rows were produced by an undeclared audio-capable pipeline, which breaks the controlled comparison, or they are mislabeled text-only results, which collapses the central claim. Table 5 doesn't rescue it: comparing Qwen-Audio's ASR mode to Qwen-Audio finetuned with ASR-EC shows improvement, but there's no text-only LoRA on the same base model, so the gain can be attributed to task finetuning alone. The \"one-shot\" mention in Section 7.2 is another small inconsistency (the table has three-shot).\n\nThe benchmark itself has softer issues: the 10% retention of CER=0 utterances is arbitrary and unexplained, there are no error bars or significance tests, and the paper includes an absurd claim that Kaldi-K1 and Kaldi-K2 are \"the only two pipelines in the history of ASR systems.\" The reference list points to a GitHub URL, but the text says data will be released on publication, so reviewers will need to confirm what is actually available.\n\nOverall: the benchmark deserves peer review, and the LoRA finetuning experiment is publishable. But the paper as submitted presents a load-bearing internal contradiction in its main comparative result. I would send it to referees with a request for major revision: clarify or remove the unexplained multimodal rows, run the proper ablation (same base model with and without audio), add statistics, and correct the overclaims. If the authors cannot resolve the Table 4 issue, the SOTA claim should be withdrawn.\n\nWho it's for: people working on Chinese ASR post-editing or LLM-based error correction will want the benchmark, once it is verified and released. I would not cite the paper for the multimodal conclusion in its current form.","headline":"A plausible first Chinese ASR error-correction benchmark, undermined by a central Table 4 inconsistency that makes the multimodal SOTA claim unverifiable as written.","tokens_in":12601,"tokens_out":3446,"would_cite":false,"duration_ms":30249,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that on the first Chinese ASR error correction benchmark, ASR-EC, a multimodal LLM that hears the audio alongside the transcript achieves the lowest character error rates, beating prompting, text-only finetuning, and the…","keywords":["ASR error correction","Chinese speech recognition","large language models","multimodal audio-text model","benchmark dataset","character error rate","LoRA finetuning"],"falsifier":"Take a third Chinese ASR system not used in the benchmark, run its transcripts through the same multimodal finetuning protocol, and compare character error rates against text-only finetuning; if the audio-based gains disappear or reverse, the central claim is an artifact of the two chosen ASR engines.","tokens_in":11512,"feed_emoji":"🎙️","tokens_out":7233,"duration_ms":59403,"temperature":0.7,"pith_summary":"The paper sets out to give Chinese ASR error correction a public testbed and to find out how large language models should be used on it. It builds ASR-EC by decoding audio from four Chinese speech corpora with two different ASR pipelines, yielding paired erroneous and correct transcripts across short and long utterances, and claims this is the first Chinese ASR error correction benchmark. Using that benchmark, it compares three LLM strategies: prompting, LoRA finetuning, and multimodal augmentation that feeds the audio together with the transcript into the model. The headline result is that prompting alone raises the error rate, finetuning helps some models, and the multimodal approach achieves the lowest character error rates, beating both the ASR baselines and text-only correction. If the benchmark is representative, the practical route for LLM-based ASR post-processing in Chinese is to give the model access to the audio signal.","feed_headline":"Audio-plus-text LLMs beat text-only fixes for Chinese ASR","feed_subtitle":"On the new ASR-EC benchmark, only models that hear the audio beat ASR baselines and text-only finetuning.","key_machinery":"The machine that carries the argument is ASR-EC, a paired dataset of audio clips, erroneous transcripts from two ASR pipelines, and correct transcripts, split into short and long utterances and annotated by substitution, deletion, and insertion errors. The method that produces the headline result is multimodal augmentation: the audio is encoded and concatenated with an error-correction instruction and the erroneous text, and a multimodal LLM is finetuned end-to-end to output the corrected text. The audio channel lets the model resolve homophone and near-homophone confusions that are invisible from text alone, which is why the paper finds audio-plus-text correction superior to both prompting and text-only finetuning.","core_discovery":"The paper's central claim is that a multimodal LLM finetuned end-to-end on ASR-EC—presented with the raw audio and the erroneous transcript jointly—corrects Chinese ASR errors better than any prompting or text-only finetuning configuration tested. The authors report that prompting zero-shot, few-shot, or multi-step consistently increases character error rate relative to the original ASR output because LLMs over-correct. LoRA finetuning of text-only LLMs recovers most of the gap but leaves the models short of the multimodal result, with Baichuan2 the strongest of the three text models. Finetuning the audio-language model on ASR-EC also improves on its out-of-the-box ASR mode, indicating the benchmark provides a useful supervised signal even for models already pretrained on audio. The authors also identify a residual floor: errors involving names and pronouns cannot be fixed without external context or prior knowledge.","pith_inferences":["Beyond the paper: because the multimodal gains come from acoustic disambiguation, production Chinese ASR pipelines could treat a finetuned audio-language model as a post-processing stage whenever the original audio is still available.","Beyond the paper: the reported name and pronoun floor suggests that coupling correction with external knowledge sources, such as contact lists or entity databases, is a concrete next test that the paper does not run.","Beyond the paper: the 10% retention cap on error-free utterances means the benchmark over-represents erroneous speech; a deployment where most transcripts are already correct would need different precision and regression metrics to judge correction value.","Beyond the paper: since only two ASR engines generated the errors, extending the benchmark with transcripts from additional architectures would test whether the multimodal advantage holds across error distributions."],"forward_implications":["Prompting a general LLM to correct ASR output without training data is counterproductive: it raises character error rate on both benchmark sets.","Parameter-efficient finetuning with LoRA turns text-only LLMs into usable correctors, but which models benefit varies and none reach the multimodal result.","Giving a multimodal LLM the audio alongside the transcript yields the largest character error rate reductions across short, long, and mixed utterances.","Finetuning an audio-language model on ASR-EC improves its correction performance over its default ASR mode, so the benchmark is useful even for audio-pretrained models.","Name and pronoun errors remain a floor that neither text nor audio context in the benchmark can fix."],"supporting_citations":[{"why":"Kaldi toolkit supplies the two ASR pipelines whose decoding errors form the benchmark's erroneous transcripts.","marker":"[2]"},{"why":"THCHS-30 contributes recorded news speech as one of the four source corpora.","marker":"[44]"},{"why":"AISHELL-1 contributes short Mandarin utterances across smart-home and other domains.","marker":"[45]"},{"why":"AISHELL-2 contributes large-scale read speech spanning voice-command and industrial domains.","marker":"[46]"},{"why":"WeNetSpeech contributes long and multi-sentence utterances, making the benchmark harder.","marker":"[47]"},{"why":"LoRA is the parameter-efficient finetuning method used in the text-only finetuning paradigm.","marker":"[6]"},{"why":"Qwen-Audio is the multimodal model that carries the audio-plus-text augmentation and finetuning experiments.","marker":"[14]"},{"why":"ChatGLM3 is one of the three Chinese-capable text LLMs evaluated under prompting and finetuning.","marker":"[48]"},{"why":"Qwen is one of the three text LLMs evaluated and the text-model comparison point.","marker":"[49]"},{"why":"Baichuan2 is the third text LLM, reported as the strongest text-only finetuned model.","marker":"[50]"}],"fun_headline_variants":["Hear the audio: multimodal LLM wins Chinese ASR error fix","Audio-aware LLM tops Chinese ASR error correction benchmark","Text-only prompts hurt ASR, audio plus text heals it","New ASR-EC benchmark: only audio-aware LLMs correct errors well","Finetune with audio, not text alone, for Chinese ASR fixes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's error mix is assumed representative because it comes from two specific ASR engines and keeps only 10% of error-free transcripts; if that mix is unrepresentative, the relative gains of the multimodal method may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Hear the audio: multimodal LLM wins Chinese ASR error fix","Audio-aware LLM tops Chinese ASR error correction benchmark","Text-only prompts hurt ASR, audio plus text heals it","New ASR-EC benchmark: only audio-aware LLMs correct errors well","Finetune with audio, not text alone, for Chinese ASR fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001004,"raw_usage":{"total_tokens":4296,"prompt_tokens":1046,"completion_tokens":3250,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":3156}},"tokens_in":662,"tokens_out":3250,"duration_ms":22089,"temperature":1.0,"reasoning_tokens":3156,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:47:54.592175+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a third Chinese ASR system not used in the benchmark, run its transcripts through the same multimodal finetuning protocol, and compare character error rates against text-only finetuning; if the audio-based gains disappear or reverse, the central claim is an artifact of the two chosen ASR engines.","supporting_citations":[{"cited_title":"The kaldi speech recognition toolkit,","cited_arxiv_id":null,"evidence_quote":"Kaldi toolkit supplies the two ASR pipelines whose decoding errors form the benchmark's erroneous transcripts."},{"cited_title":"Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition,","cited_arxiv_id":null,"evidence_quote":"WeNetSpeech contributes long and multi-sentence utterances, making the benchmark harder."},{"cited_title":"Glm: General language model pretraining with autoregressive blank infilling,","cited_arxiv_id":null,"evidence_quote":"ChatGLM3 is one of the three Chinese-capable text LLMs evaluated under prompting and finetuning."}],"review_version":1}