{"id":"9f54761d-015e-483c-a331-34a54cce9c9e","arxiv_id":"2608.05759","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Dedicated context biasing methods cut rare-word error rates by up to 88 percent relative and stayed robust to distractor lists, while speech LLMs were strong on read speech but degraded when prompts grew or were reordered.","lead":"This paper compares two ways to help speech recognition systems recognize rare words: giving the model a list of words to bias toward, or prompting a speech large language model with context. Across read and conversational audio, the word-list methods were more reliable, while speech LLMs were strong on read audio but sensitive to how the prompt was written.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Speech-LLM comparison depends on one arbitrary prompt ordering; the paper asserts word-order sensitivity but gives no ordering experiment, so the robustness gap may be an artifact.","rationale":"The reader's weakest assumption already identifies alphabetical sorting as potentially unfair. My review agrees and sharpens it: the paper not only uses one ordering but also asserts order sensitivity without reporting the measurement. This matters because the headline trade-off—robust context biasing versus flexible but prompt-sensitive speech LLMs—is exactly what the comparison is supposed to establish. If the speech-LLM results depend on an unexamined prompt-construction detail, the conclusion is not yet fully supported. Other weaknesses (no error bars, no released code, hand-chosen test sets) are real but secondary; they would not by themselves overturn the large WER differences. The proposed test is cheap and directly settles whether the concern lands. Since the current verdict is CONDITIONAL and the appropriate response is to require such an experiment before relying on the conclusion, no change to the reader's verdict is needed.","tokens_in":12035,"tokens_out":7336,"duration_ms":65474,"concrete_test":"Re-run the speech-LLM rows of Table I on Earnings-21 and Yodas at N=0, 10, and 250 with the prompt words ordered (i) alphabetically as in the paper, (ii) by first occurrence in the reference transcript, and (iii) with target words placed at the prompt start before distractors, plus a random-order average over 5 seeds. If the best of these orderings brings any of C/D/E's N=250 BWER within 10% relative of method B on both non-read sets, the alphabetical-ordering choice is load-bearing and the robustness conclusion needs revision; if no ordering does, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central conclusion—dedicated context biasing remains the more robust choice—rests on the speech-LLM columns of Table I. Those columns are all produced with a single prompt template: \"we sort the words in the prompt alphabetically for all speech LLMs\" (Section IV-A). The same paragraph claims \"We found that they are sensitive with respect to the order in which the words from the context biasing list are placed in the prompt,\" yet no experiment, table, or number supports this assertion. Because ordering is known to affect these models, alphabetical order is not a neutral choice: it arbitrarily favors words early in the alphabet and may systematically depress aggregate BWER. Context biasing methods A and B are order-agnostic, so the comparison is between a method whose prompt interface is fixed in its best configuration and speech LLMs evaluated under one possibly poor configuration. If an alternative ordering (e.g., words in reference order, or target words placed at the start of the prompt) improves speech-LLM BWER on Earnings-21 or Yodas enough to close or reverse the gap at N=250, the paper's main claim would not survive. The authors' limitations section concedes free-form prompts are not tested, but the word-list setting is exactly the setting where prompt order can be optimized without changing the interface.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript compares two context biasing extensions of Whisper (methods A and B) with three speech LLMs (Qwen3-ASR, Qwen3-Omni, VibeVoice-ASR) on rare-word recognition. It reports biased, unbiased, and overall word error rates (BWER, UWER, WER) on Earnings-21, LibriSpeech test-clean/test-other, and Yodas, with 0, 10, 100, and 250 distractor words. It also evaluates a filtering stage that uses Qwen3-Omni to prune the bias list. The headline claims are that context biasing cuts biased WER by up to 88% relative, leaves other words largely unaffected, and remains robust to distractors, while speech LLMs are strong on read speech but weaker on non-read speech and sensitive to distractor count and prompt word order.","tokens_in":12494,"tokens_out":6074,"duration_ms":52568,"significance":"If the results hold, the paper offers practical guidance for selecting between dedicated context biasing and speech-LLM prompting. The strengths are the controlled distractor-count sweep, the use of three public test sets covering read and non-read speech, the clear metric definitions, and the filtering study with recall/precision reporting. The paper is honest about limitations, including the computational cost of filtering and the fact that free-form prompting is not tested. The main concern is that the speech-LLM comparison rests on a single prompt-ordering choice that is not experimentally justified, and no statistical uncertainty is reported for the key comparisons.","major_comments":[{"comment":"The speech-LLM columns are all produced with alphabetical sorting of the prompt words, while the paper asserts without supporting experiment that speech LLMs are order-sensitive and perform better when a word is at the beginning of the prompt. Since the central conclusion that context biasing is more robust than speech LLMs depends on these columns, the missing order experiment is load-bearing. Please add a controlled comparison of at least two alternative orderings (e.g., random order and relevant-word-first) on Earnings-21 and Yodas at N=0 and N=250, and report the resulting BWER values.","section":"Section IV-A and Table I"},{"comment":"No confidence intervals, significance tests, or repeated distractor draws are provided. Claims such as “method B is better than all speech LLMs” at N=10 and “UWER essentially unchanged” are based on single test-set numbers, and the distractors are described as “chosen randomly” without a seed or repeated sampling. Add bootstrap confidence intervals over utterances and/or multiple distractor draws so the reader can assess whether the reported gaps are significant.","section":"Section V and Table I"},{"comment":"The text states that for models C, D, and E “the BWER performance without relevant context does not change significantly when adding distractors,” but the table shows relative changes of roughly 5–15% (for example, Earnings-21, model D: 30.81 at N=0 vs. 26.29 at N=250). Without a statistical test, “significantly” is unsupported; rephrase as a magnitude statement or add significance tests.","section":"Section V, Table I, earnings and Yodas rows"}],"minor_comments":[{"comment":"There are typographical issues: “VibeV oice-ASR” has an unintended space, “context biassing” appears in Section V, and Table I mixes commas and periods as decimal separators.","section":"Throughout"},{"comment":"The sentence “We found that they are sensitive with respect to the order in which the words from the context biasing list are placed in the prompt” reports a finding without any accompanying data; either provide the supporting experiment or rephrase it as a hypothesis.","section":"Section IV-A"},{"comment":"The introduction lists “quantifying sensitivity to distractors and prompt word order” as a contribution, but no prompt-order result appears in Section V; align the stated contributions with the experiments actually reported.","section":"Section I and Section V"}],"recommendation":"major_revision","confidential_remarks":"The missing prompt-order experiment is the key obstacle: the speech-LLM columns are produced under an arbitrary alphabetical ordering, and the paper explicitly asserts order sensitivity without data. I would not accept the paper in its current form without an order-sensitivity experiment and some form of uncertainty quantification for the headline comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read. The paper does something most prior work skips: a controlled, head-to-head comparison of two Whisper-based context biasing methods against three speech LLMs on read and non-read speech, with BWER, UWER, and WER. The distractor-count sensitivity result is new and practically relevant—method B cuts biased WER by up to 88% relative, and speech LLMs degrade steeply as distractors accumulate. The filtering experiment is a nice addition, and the finding that speech LLMs still trail method B after filtering on non-read speech strengthens the main conclusion. Credit where due: the evaluation is careful, the test sets are appropriate, and the limitations section is honest about the word-list-only scope.\n\nThe soft spot is real. Section IV-A states that speech LLMs are sensitive to prompt word order and that alphabetical sorting was used for fairness, but no experiment, table, or number supports that assertion. The paper says words at the beginning of the prompt perform better, so alphabetical order is not a neutral choice—it may systematically disadvantage some words. The stress-test concern is legitimate: if another ordering closes the gap at N=250, the central trade-off claim would need qualification. That said, I don't think this kills the paper. The distractor effect is large, monotonic, and consistent across three speech LLMs and three test sets, and the filtering stage—which also uses alphabetical order—still leaves a gap. Still, the unsupported order-sensitivity claim is a load-bearing piece of the argument, and a reviewer should push for either an ordering experiment or a clear statement that the comparison is conditional on one arbitrary ordering.\n\nOther weaknesses are proportional: no confidence intervals or significance tests, which matters in a comparison paper, though the effect sizes are large. No code, data, or prompt templates released, which is especially annoying given the prompt-order issue. The Yodas training-inference mismatch explanation is speculative but clearly labeled.\n\nWho is this for? ASR practitioners choosing between context biasing and speech LLMs for rare-word recognition. They'll get a useful rule of thumb, but they should treat the prompt-sensitivity claim as provisional. This deserves a serious referee, not a desk reject, but it needs a major revision or a strong rebuttal on the ordering point before I'd fully trust the headline conclusion.","headline":"A genuinely useful empirical comparison, but the paper asserts prompt-order sensitivity without showing the data, and its alphabetical ordering could be loading the dice against speech LLMs.","tokens_in":12841,"tokens_out":2117,"would_cite":false,"duration_ms":21112,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When both strategies can take the same word list, dedicated context biasing is the more reliable way to recognize new and rare words, especially on non-read speech.","keywords":["context biasing","speech LLMs","rare word recognition","automatic speech recognition","word error rate","prompt sensitivity","distractor robustness"],"falsifier":"Re-run the Earnings-21 and Yodas experiments with speech-LLM prompts ordered by placing the target word first instead of alphabetically; if a natural ordering brings their biased WER at 250 distractors down to or below method B's, the paper's robustness conclusion is falsified.","tokens_in":11871,"feed_emoji":"🎙️","tokens_out":6225,"duration_ms":49560,"temperature":0.7,"pith_summary":"The paper sets out to decide which of two ways to help an ASR system recognize new and rare words—names, acronyms, domain terms—works better in practice: dedicated context biasing, where a word list is fed into a modified Whisper model, or speech large language models prompted with the same list. It claims that the dedicated methods cut the error rate on the biased words by up to 88% relative while leaving other words largely unaffected, and that they tolerate noisy, long word lists far better than speech LLMs do. The speech LLMs, it argues, are strong on read speech but generalize less well to conversations and meetings, and their accuracy depends on how many distractor words share the prompt and in what order the words appear. This matters because rare words usually carry the informative content of an utterance, so knowing which technique can actually write them correctly determines whether a transcript is usable.","feed_headline":"Context biasing beats speech LLMs on new-word accuracy","feed_subtitle":"When a clean word list is available, biasing holds up under noise; speech LLMs depend on prompt order.","key_machinery":"The load-bearing object is the context biasing list $Z=(Z_1,\\ldots,Z_L)$ supplied at inference. Method A adds a context-attention layer after each decoder cross-attention layer: for each decoded token it computes similarity between the decoder state and summary vectors of the list entries, and with an $\\mathrm{argmax}$ selects the single most relevant entry to attend to; if none is relevant, it uses a learned dummy vector. Method B extends the decoder's vocabulary with dynamic tokens $v_l$ for list entries, embedding them from the same summary vectors and scoring them with a scaled dot product between the decoder output and the context summaries. The speech LLMs instead receive the words as text tokens in the prompt, which is why prompt order and list length affect them.","core_discovery":"On its own terms, the paper's discovery is a trade-off map rather than a single winner. Against Whisper large-v2, the two context biasing methods improve biased WER by 37–70% relative for method A and 48–88% relative for method B when the correct word is in the list, while unbiased WER moves by at most a few percent. Method B is the best non-LLM overall, but loses ground as distractors are added because it wrongly inserts dynamic vocabulary tokens. The three speech LLMs are competitive or better on read LibriSpeech—Qwen3-Omni reaches 1.1% biased WER on test-clean—but fall behind on Earnings-21 and Yodas, and their biased WER degrades by 39% to 570% relative when 250 distractors are added. Filtering the word list with an auxiliary speech LLM recovers much of the loss but does not close the gap on non-read speech.","pith_inferences":["A practical extension the paper does not test: use the positional sensitivity itself as a lever—place the most important or hardest words first in the prompt, or order by acoustic confusability—and measure whether speech LLMs close the gap on non-read speech.","Because the paper intentionally isolates word-list prompting, its verdict on speech LLM flexibility is limited to that interface; in settings where context can be a free-form description, the comparison may look different, and the paper notes this capability but does not measure it.","A testable prediction follows from the distractor results: speech-LLM biased WER should degrade monotonically with prompt length and with later word position, while context biasing should stay roughly flat; a practitioner could build a cheap pre-filter that orders prompts by estimated target probability and verify the predicted gap."],"forward_implications":["With a clean, short word list, dedicated context biasing—especially method B—attains lower biased WER on non-read speech than any of the three speech LLMs tested.","Adding up to 250 unrelated distractor words costs context biasing at most a few percent relative, while it can multiply speech-LLM biased WER several-fold; applications with long lists should expect the gap to widen.","Speech LLMs remain attractive for read-style audio where their unbiased baselines are already strong; Qwen3-Omni's 1.1% biased WER on LibriSpeech test-clean is the best single number in the table.","Filtering the list with an auxiliary model restores most of the speech-LLM loss at high recall, but leaves them behind method B on Earnings-21 and Yodas, so filtering is a partial mitigation, not a fix.","Since speech LLMs are word-order sensitive, any deployment that feeds a list must control prompt order; alphabetical sorting is the paper's controlled choice, not an instruction for practice."],"supporting_citations":[{"why":"Whisper large-v2 is the baseline ASR model that both context biasing methods extend, so its error rates are the reference for all improvements.","marker":"[3]"},{"why":"Supplies the training scheme (context list sampled from labels) used to train both context biasing models.","marker":"[12]"},{"why":"Provides architecture B (dynamic-vocabulary extension) and is the closest prior comparison the paper extends to speech LLMs and non-read speech.","marker":"[21]"},{"why":"Defines biased and unbiased WER and the rare-word selection procedure reused for LibriSpeech and metric reporting.","marker":"[13]"},{"why":"Qwen3-ASR is one of the three compared speech LLMs and its claim of context-token training motivates the prompt-based interface.","marker":"[27]"},{"why":"Qwen3-Omni is the largest compared speech LLM and the auxiliary model used to filter the bias list.","marker":"[28]"},{"why":"VibeVoice-ASR is the third compared speech LLM, supplying the best distractor-free biased WER on Earnings-21 and Yodas.","marker":"[29]"},{"why":"Earnings-21 is one of the two non-read test sets used to measure generalization beyond read speech.","marker":"[32]"},{"why":"LibriSpeech provides the read-speech test sets and the rare-word extraction rule.","marker":"[33]"},{"why":"Yodas is the second non-read, YouTube-based test set used for the rare-word evaluation.","marker":"[34]"}],"fun_headline_variants":["Context biasing slashes new-word errors by up to 88%","Speech LLMs lose ground on non-read speech","Word list noise hits speech LLMs hard","Biasing vs LLMs: a trade-off for new words"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison depends on treating alphabetically sorted prompts as the representative way to prompt speech LLMs; if some natural ordering systematically improves their rare-word accuracy, the conclusion that dedicated context biasing is more robust could change.","fun_headline_variants_meta":{"raw":{"variants":["Context biasing slashes new-word errors by up to 88%","Speech LLMs lose ground on non-read speech","Word list noise hits speech LLMs hard","Biasing vs LLMs: a trade-off for new words"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000386,"raw_usage":{"total_tokens":2018,"prompt_tokens":900,"completion_tokens":1118,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":1052}},"tokens_in":516,"tokens_out":1118,"duration_ms":9109,"temperature":1.0,"reasoning_tokens":1052,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:34:19.351219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Earnings-21 and Yodas experiments with speech-LLM prompts ordered by placing the target word first instead of alphabetically; if a natural ordering brings their biased WER at 250 distractors down to or below method B's, the paper's robustness conclusion is falsified.","supporting_citations":[{"cited_title":"Instant one-shot word- learning for context-specific neural sequence-to-sequence speech recog- nition,","cited_arxiv_id":null,"evidence_quote":"Supplies the training scheme (context list sampled from labels) used to train both context biasing models."},{"cited_title":"Yodas: Youtube-oriented dataset for audio and speech,","cited_arxiv_id":null,"evidence_quote":"Yodas is the second non-read, YouTube-based test set used for the rare-word evaluation."}],"review_version":2}