REVIEW 3 major objections 5 minor 38 references
Improving Contextual ASR via Multi-grained Fusion with Large Language Models
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A multi-grained fusion method that jointly uses token-level and phrase-level scores from ASR and LLM improves keyword recognition in contextual ASR.
desk verdict A genuinely new multi-grained fusion idea for contextual ASR, but the training objective doesn't match the inference-time joint distribution, which is a real gap. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
The authors test on Chinese and English datasets with keyword lists. They report lower errors on the keywords than previous methods, and about the same error rate on ordinary words. However, the training procedure optimizes the token level and phrase level separately, while the final decoding uses their product. The paper never trains the product directly, which is a logical gap. Several implementation details are missing, including how the fake keyword is represented and how beam search handles whole phrases. No code is released.
If the method holds up, it offers a lightweight way to inject LLM knowledge into ASR without fine-tuning the LLM, useful for assistants that must recognize contact names or domain-specific vocabulary.
Extended reading notes
Core claim
The central claim is in the abstract: 'experiments on Chinese and English datasets demonstrate that our approach achieves state-of-the-art performance on keyword-related metrics while preserving high accuracy on non-keyword text.' Concretely, Table 1 shows B-CER of 2.2 vs CopyNE's 3.4 on Aishell, and Table 2 shows B-WER of 5.36 vs MaLa-ASR's 5.47 on Slidespeech.
Load-bearing premise
The training objective (Eq. 11-13) minimizes Ltok over ptok and Lphr over pphr independently, but inference (Eq. 10) uses pjoi = pphr(k0)*ptok for token decisions and pphr(ki) for phrase decisions. The paper never defines a loss over pjoi, so no gradient directly rewards high joint probabilities; the model may be miscalibrated at test time. This mismatch is in Section 3.4, Eq. 10 and Section 4.2.1, Eq. 11-13.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a multi-grained fusion approach for contextual ASR that combines token-level and phrase-level fusion with an LLM. Whisper serves as the ASR backbone; Qwen2 or Phi-3.5 serves as a frozen LLM; a keyword prompt is injected into the LLM. Token-level fusion interpolates ASR and LLM logits using an ASR-uncertainty weight, while phrase-level fusion uses an LSTM keyword encoder and dot-product attention to score whole keywords. A fake keyword k0 unifies the two granularities in a joint distribution pjoi (Eq. 10), and beam search decodes over the union of token vocabulary and keyword list. The model is trained with a sum of token-level and phrase-level NLL losses (Eqs. 11-13). Experiments on Aishell, DC, ICI (Chinese) and Slidespeech (English) report improvements in keyword-related metrics (B-CER/B-WER, Recall) over CopyNE and MaLa-ASR, with ablations showing that both fusion modules contribute. The code and models are claimed to be public, though no working link is provided.
Significance. If the central claim holds, the paper makes a useful empirical contribution: it demonstrates that token-level and phrase-level fusion can be combined in a single late-fusion ASR framework, and the ablations suggest genuinely complementary roles for the two granularities. The method is also efficient relative to MaLa-ASR (3.8B vs 7B parameters) while matching or slightly improving keyword metrics. The authors build directly on their own CopyNE and RWCS-NER work, but the joint formulation is not circular: it is evaluated against external baselines and does not reduce to a fitted constant. However, the significance is tempered by the training/inference mismatch described below, which undermines the principled claim that the model optimizes the distribution actually used at inference. The paper also promises public code but currently points to a placeholder URL.
major comments (3)
- [Section 3.4 (Eq. 10) and Section 4.2.1 (Eqs. 11-13)] There is a train/inference mismatch: the total loss L = Ltok + Lphr optimizes ptok and pphr separately, but inference uses pjoi(z) = pphr(k0)ptok(z) for vocabulary tokens and pjoi(k) = pphr(k) for keywords. No term in Eqs. (11)-(13) is the negative log-likelihood of the gold action sequence under pjoi. In particular, the decision between copying a whole keyword and generating its first token is governed by comparing pphr(kj) with pphr(k0)ptok(first token), yet no gradient directly calibrates this comparison. For a multi-token keyword, the maximum-matching alignment in Section 4.2.1 labels interior token positions as k0, so Lphr imposes terms -log pphr(k0) at those positions and Ltok forces token-level generation inside a phrase that inference would copy as a single action. This is a load-bearing issue for the central claim that the multi-grained fusion is jointly optimized; please either derive a proper training objective over pjoi or provide evidence (e.g., comparison against training with a joint NLL) that the mismatch does not hurt the reported gains.
- [Section 3.4 (Eq. 9) and Section 4.2.1 (maximum matching)] The fake keyword k0 is not fully specified. Equation (9) changes the softmax denominator to include k0, but the paper never defines the representation r0 for k0, how it is initialized, or whether it is trained along with the keyword encoder. Without this, equation (9) is not reproducible. In addition, the maximum-matching algorithm that produces the phrase sequence P needs an exact description: for overlapping keywords, for partial matches, and for multi-token keywords it must be stated which time step receives the keyword label and which receives k0. This is also needed to assess the claimed loss behavior in the previous comment.
- [Table 2 and Section 4.4.2] The English 'state-of-the-art' claim rests on a very small difference: B-WER of 5.36 vs MaLa-ASR's 5.47 (0.11 absolute), while WER and U-WER are identical (9.14 and 9.42). No statistical significance tests, confidence intervals, or multiple-run variance are reported, so this could be within run-to-run noise. Moreover, the MaLa-ASR baseline is explicitly evaluated without LLM fine-tuning, which gives the comparison a favorable tilt. Please report variance or significance testing and clarify the training configuration of each baseline, or soften the SOTA claim to a more precise statement.
minor comments (5)
- [Section 4.2.1, Eq. (11) and Eq. (12)] Both losses are written as Ltok = -sum_t ptok(...) and Lphr = -sum_t pphr(...), which is the negative sum of probabilities, not a negative log-likelihood. The logarithm appears to be missing (it is present in the text description). Please correct the equations to -sum_t log p_tok and -sum_t log p_phr.
- [Abstract and Section 1] The abstract claims 'state-of-the-art performance', while the Introduction states performance is 'comparable to state-of-the-art methods' and 'on par with previous approaches'. Please align the claims with the actual effect sizes, especially given the English results.
- [General] The code link in the abstract and Section 5 is 'https://github.com/', which is a placeholder, not a repository. The code-availability claim cannot be verified in the current submission.
- [Figure 2] The x-axis in Figure 2 uses values 0, 50, 200, 1000 without notation for a nonlinear scale; a logarithmic scale or explicit break would make the plot easier to read. It should also clarify whether list size 0 corresponds to an empty prompt or to no keyword list.
- [References] The reference to Loshchilov et al. (2017) in the text appears as 'Loshchilov et al., 2017' while the bibliography entry has the format 'Ilya Loshchilov, Frank Hutter, et al. 2017'; please fix the author list. Similarly, the Mann et al. bibliography entry has an unusual format that should be cleaned up.
Circularity Check
No significant circularity: the joint multi-grained fusion is an empirical combination evaluated against external baselines; self-citations to CopyNE are architectural or baseline reuse, not load-bearing.
full rationale
The central claim, that jointly combining token-level and phrase-level fusion achieves state-of-the-art keyword metrics, is supported by comparisons to external systems, especially MaLa-ASR in Table 2, and to the fine-tuned Whisper baseline in Table 1. Those results are measured, not assumed, so the claim does not reduce to its inputs. The self-citations concern architectural reuse (the 3-layer LSTM keyword encoder 'Following CopyNE (Zhou et al., 2024b)'), the CopyNE baseline, the RWCS-NER test set, and metric conventions; none of these supplies an unverified premise that the paper's conclusion depends on. The joint probability in Eq. 10 is a genuine factorization, pjoi(token) = pphr(k0) * ptok(token), with both factors trained by separate losses, and the k0 prior is learned rather than fitted to test labels. The reviewer-flagged mismatch between Eq. 13, which sums Ltok and Lphr, and Eq. 10, which uses pjoi at inference, is a training/inference calibration gap, not a circular reduction: the paper does not relabel a fitted quantity as a prediction, import a uniqueness theorem, or define pjoi as the training objective. This concern is a correctness risk, not a circularity step. Overall score 1 reflects one minor non-load-bearing self-citation and otherwise independent empirical content.
Assumptions & free parameters
free parameters (1)
- relative logit weight (implicit 1.0) in Eq. 3 =
1.0 (fixed by design)
assumptions (4)
- domain assumption ASR and LLM logits can be combined additively as st = sa_t + sigmoid(ua_t)*sl_t after a fixed scalar weight.
- domain assumption The entropy of the ASR output distribution is a reliable confidence signal for weighting the LLM.
- domain assumption Replacing the ASR tokenizer with the LLM tokenizer preserves enough acoustic alignment for joint decoding.
- domain assumption A frozen pre-trained LLM, prompted with a keyword list, produces logits useful for ASR even though it has never seen speech.
invented entities (1)
-
fake keyword k0
Cite this review
Pith. "Pith review of Improving Contextual ASR via Multi-grained Fusion with Large Language Models." pith.science (2026). https://pith.science/paper/YYALUACA
@misc{pith2026250712252,
author = {Pith},
title = {Pith review of: Improving Contextual ASR via Multi-grained Fusion with Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/YYALUACA}},
note = {Machine review of arXiv:2507.12252}
}
read the original abstract
While end-to-end Automatic Speech Recognition (ASR) models have shown impressive performance in transcribing general speech, they often struggle to accurately recognize contextually relevant keywords, such as proper nouns or user-specific entities. Previous approaches have explored leveraging keyword dictionaries in the textual modality to improve keyword recognition, either through token-level fusion that guides token-by-token generation or phrase-level fusion that enables direct copying of keyword phrases. However, these methods operate at different granularities and have their own limitations. In this paper, we propose a novel multi-grained fusion approach that jointly leverages the strengths of both token-level and phrase-level fusion with Large Language Models (LLMs). Our approach incorporates a late-fusion strategy that elegantly combines ASR's acoustic information with LLM's rich contextual knowledge, balancing fine-grained token precision with holistic phrase-level understanding. Experiments on Chinese and English datasets demonstrate that our approach achieves state-of-the-art performance on keyword-related metrics while preserving high accuracy on non-keyword text. Ablation studies further confirm that the token-level and phrase-level components both contribute significantly to the performance gains, complementing each other in our joint multi-grained framework. The code and models will be publicly available at https://github.com/.
Figures
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219
arXiv 2024
-
[4]
Uri Alon, Golan Pundak, and Tara N Sainath. 2019. Contextual speech recognition with difficult negative training examples. In 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6440--6444
work page 2019
-
[5]
Ye Bai, Jingping Chen, Jitong Chen, Wei Chen, Zhuo Chen, Chuang Ding, Linhao Dong, Qianqian Dong, Yujiao Du, Kepan Gao, et al. 2024. Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition. arXiv preprint arXiv:2407.04675
arXiv 2024
-
[6]
Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In 2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA), pages 1--5
2017
-
[7]
Boli Chen, Guangwei Xu, Xiaobin Wang, Pengjun Xie, Meishan Zhang, and Fei Huang. 2022. Aishell-ner: Named entity recognition from chinese speech. In 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8352--8356
2022
-
[8]
Chen Chen, Ruizhe Li, Yuchen Hu, Sabato Marco Siniscalchi, Pin-Yu Chen, EngSiong Chng, and Chao-Han Huck Yang. 2024. It's never too late: Fusing acoustic information into large language models for automatic speech recognition. In The Twelfth International Conference on Learning Representations
work page 2024
Show all 38 references
-
[9]
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6
2023
-
[10]
Jan Chorowski and Navdeep Jaitly. 2016. Towards better decoding and language model integration in sequence to sequence models. arXiv preprint arXiv:1612.02695
2016 arXiv
-
[11]
Strimel, Ross McGowan, and Athanasios Mouchtaris
Xuandi Fu, Kanthashree Mysore Sathyendra, Ankur Gandhe, Jing Liu, Grant P. Strimel, Ross McGowan, and Athanasios Mouchtaris. 2023. https://doi.org/10.1109/ICASSP49357.2023.10094808 Robust acoustic and semantic contextual biasing in neural transducers for speech recognition . I...
2023
-
[12]
Sanchit Gandhi, Patrick von Platen, and Alexander M Rush. 2023. Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling. arXiv preprint arXiv:2311.00430
2023 arXiv
-
[13]
Yuchen Hu, Chen Chen, Chao-han Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen, and Ensiong Chng. 2024. Large language models are efficient learners of noise-robust speech recognition. In International Conference on Learning Representations
2024
-
[14]
Kaixun Huang, Ao Zhang, Zhanheng Yang, Pengcheng Guo, Bingshen Mu, Tianyi Xu, and Lei Xie. 2023. https://doi.org/10.21437/Interspeech.2023-767 Contextualized end-to-end speech recognition with contextual phrase prediction network . In 24th Annual Conference of the Internationa...
2023 doi
-
[15]
Sai Muralidhar Jayanthi, Devang Kulshreshtha, Saket Dingliwal, Srikanth Ronanki, and Sravan Bodapati. 2023. Retrieve and copy: Scaling asr personalization to large catalogs. arXiv preprint arXiv:2311.08402
2023 arXiv
-
[16]
Gonzalez, Hao Zhang, and Ion Stoica
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...
2023
-
[17]
Egor Lakomkin, Chunyang Wu, Yassir Fathullah, Ozlem Kalinli, Michael L Seltzer, and Christian Fuegen. 2024. End-to-end speech recognition contextualization with large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing ...
2024
-
[18]
Yuang Li, Yu Wu, Jinyu Li, and Shujie Liu. 2023. Prompting large language models for zero-shot domain adaptation in speech recognition. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1--8. IEEE
2023
-
[19]
Ilya Loshchilov, Frank Hutter, et al. 2017. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5
2017 arXiv
-
[20]
Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 1:3
2020 arXiv
-
[21]
OpenAI. 2023. https://arxiv.org/abs/2303.08774 GPT-4 technical report . ArXiv preprint, abs/2303.08774
2023 arXiv
-
[22]
Golan Pundak, Tara N Sainath, Rohit Prabhavalkar, Anjuli Kannan, and Ding Zhao. 2018. Deep context: end-to-end contextual speech recognition. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 418--425
2018
-
[23]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356
2022 arXiv
-
[24]
Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. 2017. Cold fusion: Training seq2seq models together with language models. arXiv preprint arXiv:1708.06426
2017 arXiv
-
[25]
Yui Sudo, Yosuke Fukumoto, Muhammad Shakeel, Yifan Peng, and Shinji Watanabe. 2024. Contextualized automatic speech recognition with dynamic vocabulary. arXiv preprint arXiv:2405.13344
2024 arXiv
-
[26]
Chuanneng Sun, Zeeshan Ahmed, Yingyi Ma, Zhe Liu, Lucas Kabela, Yutong Pang, and Ozlem Kalinli. 2024. Contextual biasing of named-entities with large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1...
2024
-
[27]
Guangzhi Sun, Chao Zhang, and Philip C Woodland. 2022. Tree-constrained pointer generator with graph neural network encodings for contextual speech recognition. arXiv preprint arXiv:2207.00857
2022 arXiv
-
[28]
Guangzhi Sun, Xianrui Zheng, Chao Zhang, and Philip C Woodland. 2023. Can contextual biasing remain effective with whisper and gpt-2? arXiv preprint arXiv:2306.01942
2023 arXiv
-
[29]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288
2023 arXiv
-
[30]
Haoxu Wang, Fan Yu, Xian Shi, Yuezhang Wang, Shiliang Zhang, and Ming Li. 2024. Slidespeech: A large scale slide-enriched audio-visual corpus. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11076--11080. IEEE
2024
-
[31]
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305
2023 arXiv
-
[32]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024 a . Qwen2 technical report. arXiv preprint arXiv:2407.10671
2024 arXiv
-
[33]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024 b . Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115
2024 arXiv
-
[34]
Guanrou Yang, Ziyang Ma, Fan Yu, Zhifu Gao, Shiliang Zhang, and Xie Chen. 2024 c . Mala-asr: Multimedia-assisted llm-based asr. arXiv preprint arXiv:2406.05839
2024 arXiv
-
[35]
Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, Fan Yu, Chao Yang, Zhendong Peng, Xiaoyu Chen, Lei Xie, and Xin Lei. 2021. Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit. In Proc. Interspeech, Brno, Czech Republic. IEEE
2021
-
[36]
Fan Yu, Haoxu Wang, Xian Shi, and Shiliang Zhang. 2024. Lcb-net: Long-context biasing for audio-visual speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024 , pages 10621--10625. IEEE
2024
-
[37]
Shilin Zhou, Zhenghua Li, Chen Gong, Lei Zhang, Yu Hong, and Min Zhang. 2024 a . https://doi.org/10.18653/v1/2024.findings-acl.111 C hinese spoken named entity recognition in real-world scenarios: Dataset and approaches . In Findings of the Association for Computational Lingui...
2024 doi
-
[38]
Shilin Zhou, Zhenghua Li, Yu Hong, Min Zhang, Zhefeng Wang, and Baoxing Huai. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.147 C opy NE : Better contextual ASR by copying named entities . In Proceedings of the 62nd Annual Meeting of the Association for Computational Ling...
2024 doi
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.