{"id":"4943640d-cf9d-4dc9-8726-c9d5b6790145","arxiv_id":"2501.09258","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Delayed fusion scores partial ASR hypotheses with a pre-trained LLM only after pruning and at word boundaries, giving lower word error rates than N-best rescoring without retraining the ASR model.","lead":"Speech recognizers can use large language models to improve accuracy, but LLMs are slow and use different word vocabularies. This paper introduces delayed fusion, which asks the LLM to score only short, pruned partial transcriptions at word boundaries, making the language model fast enough to run during real-time decoding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Re-tokenization uniqueness is load-bearing: if a word's LLM tokenization depends on neighboring words, the cached prefix score in Sec. III-B is not the probability of the actual hypothesis and beam pruning may be biased; the paper never verifies this for OpenLLaMA/Mistral.","rationale":"The reader's weakest assumption is the same load-bearing point I identify: the incremental LLM score in delayed fusion is only faithful if re-tokenizing a word in isolation yields the same token sequence as it would receive in the full hypothesis, and if the cached prefix scores remain valid for all hypotheses after pruning. This is a genuine condition for the central claim that delayed fusion 'provides lower WERs than N-best LLM rescoring and standard NLM fusion.' If the condition fails, the scores used in beam pruning and final selection are not true LLM probabilities, so the empirical WER comparisons could be compromised. The paper's own text flags the risk ('tokenization may be incorrect for incomplete hypotheses') but does not resolve it with a verification experiment or a tokenizer-level analysis for the three LLMs used. I do not see a more basic technical flaw: the algorithm is coherent, the experiments are plausible, and the streaming claim is appropriately separated in the conclusions as an advantage rather than a measured result. The missing error bars and absent code are secondary reproducibility concerns, not the most load-bearing issue. Because the correctness of the method hinges on this unverified property, the appropriate verdict remains CONDITIONAL; my stress-test does not change that verdict but sharpens the condition that should be checked before acceptance.","tokens_in":9387,"tokens_out":13753,"duration_ms":151539,"concrete_test":"Sample 1,000 decoded hypotheses from lh-clean. For each complete hypothesis string, compare (a) the delayed-fusion incremental LLM score obtained by per-word re-tokenization and cached prefixes exactly as in Sec. III-B, against (b) an exact LLM score obtained by tokenizing the full hypothesis string with the LLM tokenizer and running the LLM once. Compute the per-hypothesis log-probability differences and the fraction of utterances where re-ranking the final beam by the exact score changes the selected hypothesis. If the median absolute difference exceeds 0.1 nats, or if the selected hypothesis changes on more than 2% of utterances, the approximate score is biasing decoding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism in Section III-B depends on the assertion that with a standard SentencePiece tokenizer, \"each word is tokenized into a unique token sequence.\" Delayed fusion re-tokenizes each detected word in isolation, concatenates the per-word LLM tokenizations, and reuses cached scores for the common prefix when computing S_LM(h) ≈ log P(v_1,...,v_\\hat M) + log P(v_{\\hat M+1},...,v_{M_new}|v_1,...,v_\\hat M). This is valid only if the per-word tokenizations concatenate exactly to the tokenization of the full hypothesis prefix. That property is not guaranteed by SentencePiece (it depends on pretokenization and whether tokens can span whitespace), and it is even less automatic for byte-level BPE tokenizers such as Mistral's, where merges can cross the space boundary. If the property fails, the cached prefix v_1..v_\\hat M is not a prefix of the hypothesis's true LLM tokenization, so S_LM is the log-probability of a different string. Because these scores enter pruning (Algorithm 1, line 6) and final selection (line 13), an unfaithful S_LM can remove the correct hypothesis or select an incorrect one, and the reported WER improvements over N-best rescoring would not reflect a faithful early-integration score. The paper acknowledges the difficulty only by saying \"tokenization may be incorrect for incomplete hypotheses\" and then assumes it away; it never verifies the uniqueness assumption for OpenLLaMA or Mistral, nor does it quantify the approximation error represented by the '≈' sign.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'delayed fusion,' a first-pass decoding method for end-to-end ASR that integrates a pre-trained LLM by scoring partial hypotheses after beam pruning and at word boundaries. This permits using LLMs whose tokenizer/vocabulary differs from the ASR model without retraining. The method is evaluated on the LibriHeavy corpus with two decoding modes (CTC prefix beam search and joint CTC-attention decoding) and three LLMs (OpenLLaMA 3B/7B, Mistral 7B). The authors report WER and RTF values showing improvements over NLM shallow fusion and N-best LLM rescoring in several configurations, and they argue the approach is suitable for streaming applications.","tokens_in":9725,"tokens_out":13061,"duration_ms":122008,"significance":"If the claims hold, delayed fusion is a practically useful technique: it avoids retraining the ASR model or LLM to reconcile vocabularies, reuses a standard KV cache for efficient LLM scoring, and can operate in streaming-compatible first-pass decoding. The paper is clearly written and the experimental setup is substantial: three public LLMs, two decoding modes, and a 50k-hour corpus. The algorithm is presented in enough detail to be reimplemented, and the authors are transparent about the WER-RTF trade-off in fixed-interval fusion. The main risk is an unverified tokenization-assumption that underwrites the score computation, and the speed comparison with N-best rescoring is less clean than the abstract suggests.","major_comments":[{"comment":"The validity of the approximate LLM score in Section III-B depends on the claim that 'each word is tokenized into a unique token sequence' when a standard SentencePiece tokenizer is used. This property is not guaranteed for the LLMs employed in the experiments (OpenLLaMA and Mistral), especially for byte-level BPE tokenizers where merges can cross whitespace boundaries. If tokenization of a word depends on neighboring words, the concatenation of per-word tokenizations need not equal the tokenization of the full hypothesis prefix, so the cached prefix score log P(v1..v_{\\hat M_j}) is the probability of a different string. Because these scores feed directly into pruning (Algorithm 1, line 6) and final hypothesis selection (line 13), an unfaithful SLM can bias the search and compromise the reported WER comparisons. The paper itself acknowledges that 'tokenization may be incorrect for incomplete hypotheses' but does not verify that the word-boundary re-tokenization resolves this for the three LLMs. I ask the authors to empirically measure the mismatch rate between per-word concatenated tokenizations and full-sequence tokenizations on the dev set, and to quantify how often the cached prefix is not a true prefix; if the mismatch is non-negligible, the scoring procedure should be revised.","section":"III-B"},{"comment":"The abstract and the contributions list state that delayed fusion provides 'improved decoding speed ... compared to N-best rescoring.' Table II does not support this as stated: with beam size 10, N-best rescoring (N=10) has RTF 0.029, while fixed-interval delayed fusion ranges from 0.041 to 0.063 and shortest-hypothesis delayed fusion is 0.115. The only evidence for a speed advantage is Figure 3, but neither the text nor the caption reports the underlying numbers or defines how the N-best list size N scales with the beam size. To support the headline speed claim, please provide the full WER-RTF data for the beam-size sweeps in Figure 3 as a table, and qualify the claim to the configurations where delayed fusion is actually faster.","section":"IV-B (Table II, Fig. 3) and Abstract"},{"comment":"The conclusion claims 'significant WER reduction' and 'lower WERs than N-best LLM rescoring and standard NLM fusion.' The differences in Table II and Figure 3 are small (e.g., 1–3% relative WER in the Fig. 3 comparison), and the paper reports single point estimates without error bars, confidence intervals, or significance tests. With results from a single corpus (LibriHeavy) and a single ASR model family (CTC-AED), it is not possible to assess whether the improvements are stable. Please add utterance-level confidence intervals or a significance test for the main delayed-fusion vs. N-best-rescoring comparisons, or explicitly frame the observed differences as tentative rather than statistically established.","section":"V"}],"minor_comments":[{"comment":"In the abstract, 're-tokenizion' is a typo for 're-tokenization'.","section":"Abstract"},{"comment":"The score combination in Algorithm 1 and the selection rule in line 13 are written as SE2E + SLM, but Section IV-A says fusion weights are tuned on the dev set for each LM. Please define the weighted score explicitly (e.g., SE2E + λ SLM) and state how λ is used in the algorithm.","section":"Algorithm 1"},{"comment":"In the approximate score equation, \\hat M_j is used as the split point, but the text uses j both for the decoding-frame index and the LLM-call index. Please clarify the indexing convention, e.g., denote the LLM-call index by k and define \\hat M_k as the shortest sequence length at call k.","section":"III-B"},{"comment":"The description of the SentencePiece tokenizer for the ASR model does not state whether the unigram or BPE model is used or whether pretokenization is enabled. This is directly relevant to the uniqueness assumption in Section III-B, so please specify.","section":"IV-A"},{"comment":"The caption lists curves (c)–(f), but the main text does not explain what each curve represents or give the N-best list sizes used. Please add a description or a table with the plotted values.","section":"Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a speech/language-processing journal and presents a practical idea that could be useful if the technical assumptions are verified. The most important issue is the tokenization-uniqueness assumption, which is load-bearing for the score computation; I would like to see an empirical verification or an algorithmic fix. The speed claim also needs to be reconciled with Table II. I believe these are addressable within a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading if you work on E2E-ASR decoding or LLM fusion. The idea is simple and practical: score partial hypotheses with the LLM only after pruning, at word boundaries, and reuse cached KV states for common prefixes. That reduces LLM calls and handles vocabulary mismatch through re-tokenization. This is a legitimate extension of word-triggered shallow fusion; the controllable fusion interval and batched KV-cached LLM scoring are new enough to count.\n\nWhat impresses me: the paper does the right comparison. They train a CTC-AED model with the LLM tokenizer and compare delayed fusion against real LLM shallow fusion (Table III). They also compare against N-best rescoring over beam sizes and report consistent WER gains: 4-13% relative over no LM and a smaller but consistent improvement over NLM shallow fusion. The experiments use three public LLMs including Mistral, and the RTF numbers on a V100 are plausible. The authors also honestly note that their WERs cannot be strictly compared to LibriHeavy baselines, and that NLM shallow fusion followed by N-best rescoring is competitive.\n\nSoft spots, in order of importance. First, the central approximation in Section III-B relies on the claim that with a standard SentencePiece tokenizer, 'each word is tokenized into a unique token sequence.' That is not generally guaranteed and is not verified for OpenLLaMA or Mistral. If a word's tokenization depends on neighboring words, the cached prefix score is not the probability of the actual hypothesis, and beam pruning could be biased. The paper should either verify the property for the tokenizers used or quantify the approximation error. This is the one thing that could change the WER conclusions. Second, there are no error bars or significance tests; differences like 3.05 vs 3.08 are treated as meaningful. Third, the streaming claim is not backed by an experiment; all decoding uses full utterance context. Fourth, the RTF comparisons are a little tangled: N-best rescoring with small N can be faster than delayed fusion, so the speed claim should not be phrased unconditionally. Fifth, no code or configs are released, and results come from a single corpus. These are standard weaknesses for a systems paper, not fatal ones.\n\nThe citation pattern is fine; the self-citations are to the authors' prior decoding work and are relevant. There is no sign of circular fitting or invented entities.\n\nMy take: this deserves a serious referee. I would ask for the tokenization analysis and error bars before accepting, but the method and experiments are solid enough to publish after revision. I would cite it in my own work if I were writing on LLM-ASR fusion.","headline":"Delayed fusion is a real and useful extension of shallow fusion for LLM-ASR, with credible WER/RTF gains, but the paper never verifies the tokenization-uniqueness assumption that its score approximation rests on.","tokens_in":788,"tokens_out":852,"would_cite":true,"duration_ms":37601,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Delayed fusion lets LLMs cut word error rate 4-13% in decoding.","keywords":["delayed fusion","large language model","speech recognition","shallow fusion","N-best rescoring","CTC beam search","tokenization mismatch","LLM fusion"],"falsifier":"Take a set of decoded hypotheses, and for each word-boundary prefix that triggered delayed fusion, compute the LLM log-probability of that prefix alone and again with the following word included before re-tokenization; if the two log-probabilities differ by more than a small threshold on a substantial fraction of prefixes, the cached-prefix scores used for pruning are not the true conditional LM scores, and the WER comparisons would be affected.","tokens_in":9197,"feed_emoji":"🎙️","tokens_out":15104,"duration_ms":117939,"temperature":0.7,"pith_summary":"The paper introduces delayed fusion, a way to add a pretrained large language model's scores to a speech recognizer's first-pass beam search without retraining either model. Instead of scoring every partial hypothesis at every step, the decoder waits until after pruning and until a hypothesis has reached a word boundary, then re-tokenizes that boundary prefix and scores all surviving hypotheses in one batched LLM call. This keeps LLM inference calls proportional to the length of the shortest hypothesis rather than to the number of frames, and it lets the ASR and the LLM keep different token vocabularies. On the LibriHeavy corpus, delayed fusion with OpenLLaMA 3B, OpenLLaMA 7B, and Mistral 7B reports word error rates below the baseline and below N-best rescoring, at real-time factors around 0.12–0.17. The practical point: an off-the-shelf LLM can be evaluated or used in a streaming ASR pipeline without retraining.","feed_headline":"Delayed fusion lets LLMs cut word error rate 4-13% in decoding","feed_subtitle":"OpenLLaMA and Mistral cut WER at near-real-time speed, without retraining the ASR model.","key_machinery":"The load-bearing mechanism is the fusion-condition gate FUSABLE called inside the beam-search loop, together with the re-tokenization step at word boundaries. After the decoder extends and prunes the hypothesis list to the top K, FUSABLE decides whether to invoke the LLM: shortest-hypothesis fusion fires whenever the shortest re-tokenized prefix among surviving hypotheses grows, and fixed-interval fusion fires every I frames provided at least one hypothesis changed. At each LLM call, the longest prefix ending at a word boundary is re-tokenized with the LLM's SentencePiece tokenizer, and the LLM scores the newly appended tokens in one batched forward pass using a key-value cache from the previous call. This keeps the number of LLM calls at most one per token of the shortest hypothesis and makes N-best rescoring a special case when the gate never fires.","core_discovery":"Delayed fusion makes the LLM a late but cheap participant in first-pass beam search. The decoder extends and prunes hypotheses as usual; only afterward does a fusion condition decide whether to call the LLM. The shortest-hypothesis trigger fires when the shortest re-tokenized prefix among surviving hypotheses has grown, i.e. $FUSABLE(H_{0:t},t)$ is true when $\\varphi(\\bar{H}_{t-1}) < \\varphi(\\bar{H}_t)$, where $\\varphi$ is the length of the shortest sequence in the re-tokenized hypothesis list. At each trigger, every surviving hypothesis is re-tokenized at the longest prefix ending in a word-end token, and the LLM logscores the newly added tokens in one batched forward pass, reusing a key-value cache from the previous call: $S_{LM}(h) = \\log P_{LM}(\\text{prefix}) + \\log P_{LM}(\\text{new tokens} \\mid \\text{prefix})$. Because scoring happens after pruning, the number of LLM calls is bounded by the length of the shortest hypothesis, not the number of frames. On LibriHeavy, delayed fusion with OpenLLaMA 3B, OpenLLaMA 7B, and Mistral 7B reports 3.05%/5.68% WER on lh-clean/lh-other at a 0.115 real-time factor for CTC-prefix search, and the paper concludes it gives lower WERs than N-best LLM rescoring and standard NLM fusion.","pith_inferences":["Because the word-boundary uniqueness claim is specific to unigram SentencePiece tokenizers, porting delayed fusion to BPE or WordPiece tokenizers may require a stricter fusion condition to avoid context-dependent tokenizations.","The same gating idea could be applied to RNN-T decoding, where the predictor emits word pieces, allowing LLM fusion at emitted word boundaries without frame-level triggers.","The reported benefit should be interpreted primarily as a pruning-error reduction: with a far larger beam, delayed fusion's advantage over N-best rescoring may shrink, since the first-pass search would already retain the correct hypothesis."],"forward_implications":["Delayed fusion can run in streaming first-pass decoding, so live-captioning and other latency-sensitive applications can use LLM scores where N-best rescoring cannot.","The same ASR system can be evaluated against many off-the-shelf LLMs without retraining or fine-tuning, turning LLM comparison in ASR into a drop-in scoring change.","The fusion interval I sets a continuous trade-off between decoding speed and accuracy; at large I the method approaches N-best rescoring, at small I it approaches shallow fusion.","Combining delayed LLM fusion with a small in-domain NLM shallow fusion reduces pruning errors, at the cost of extra NLM computation."],"supporting_citations":[{"why":"Supplies the CTC prefix beam search and the space-token trigger for word-level LM scoring that delayed fusion adapts.","marker":"[21]"},{"why":"Prior end-to-end decoding that triggers a word-based LM at word boundaries, the direct precursor to delayed fusion's fusion condition.","marker":"[22]"},{"why":"Provides the SentencePiece tokenizer whose claimed unique token sequence per word justifies re-tokenizing partial hypotheses at word boundaries.","marker":"[30]"},{"why":"Provides the 50k-hour LibriHeavy corpus used for training and evaluation.","marker":"[31]"},{"why":"Defines the hybrid CTC/attention architecture used as the ASR model in all experiments.","marker":"[32]"},{"why":"Provides the joint CTC-attention decoding algorithm used as the label-synchronous evaluation mode.","marker":"[34]"},{"why":"Mistral 7B, one of the three off-the-shelf LLMs whose delayed-fusion scores are evaluated.","marker":"[4]"},{"why":"OpenLLaMA 3B and 7B v2, the two open LLMs used to test delayed fusion across model sizes.","marker":"[11]"}],"fun_headline_variants":["Delayed fusion: LLMs join beam search late, cut WER and cost","LLM after pruning: delayed fusion cuts calls and errors","Fuse LLMs late in beam search for faster, better ASR","Delayed fusion: LLM scoring after pruning trims WER and latency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the way the LLM tokenizer splits a word into pieces does not depend on the words around it, and that the LLM score cached for the shortest prefix stays valid for every beam hypothesis that shares that prefix.","fun_headline_variants_meta":{"raw":{"variants":["Delayed fusion: LLMs join beam search late, cut WER and cost","LLM after pruning: delayed fusion cuts calls and errors","Fuse LLMs late in beam search for faster, better ASR","Delayed fusion: LLM scoring after pruning trims WER and latency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000417,"raw_usage":{"total_tokens":2234,"prompt_tokens":1112,"completion_tokens":1122,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":728,"completion_tokens_details":{"reasoning_tokens":1043}},"tokens_in":728,"tokens_out":1122,"duration_ms":8639,"temperature":1.0,"reasoning_tokens":1043,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:06:46.212675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of decoded hypotheses, and for each word-boundary prefix that triggered delayed fusion, compute the LLM log-probability of that prefix alone and again with the following word included before re-tokenization; if the two log-probabilities differ by more than a small threshold on a substantial fraction of prefixes, the cached-prefix scores used for pruning are not the true conditional LM scores, and the WER comparisons would be affected.","supporting_citations":[{"cited_title":"End-to-end speech recognition with word-based RNN language models,","cited_arxiv_id":null,"evidence_quote":"Prior end-to-end decoding that triggers a word-based LM at word boundaries, the direct precursor to delayed fusion's fusion condition."},{"cited_title":"Sentencepiece: A simple and language inde- pendent subword tokenizer and detokenizer for neural text processing,","cited_arxiv_id":null,"evidence_quote":"Provides the SentencePiece tokenizer whose claimed unique token sequence per word justifies re-tokenizing partial hypotheses at word boundaries."},{"cited_title":"Libriheavy: a 50,000 hours asr corpus with punctuation casing and context,","cited_arxiv_id":null,"evidence_quote":"Provides the 50k-hour LibriHeavy corpus used for training and evaluation."},{"cited_title":"Hybrid CTC/attention architecture for end-to-end speech recognition,","cited_arxiv_id":null,"evidence_quote":"Defines the hybrid CTC/attention architecture used as the ASR model in all experiments."},{"cited_title":"Joint CTC/attention decoding for end-to-end speech recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the joint CTC-attention decoding algorithm used as the label-synchronous evaluation mode."},{"cited_title":"OpenLLaMA: An open reproduction of LLaMA,","cited_arxiv_id":null,"evidence_quote":"OpenLLaMA 3B and 7B v2, the two open LLMs used to test delayed fusion across model sizes."}],"review_version":1}