{"id":"ff9a4b87-50dc-48bc-927f-d46fc14f22e2","arxiv_id":"1908.02914","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ASR word confidence features and forced decoding of unknown words improve neural question answering accuracy under noisy speech input.","lead":"This paper trains question answering models on text distorted by automatic speech recognition and shows that using the recognizer's word-level confidence scores, plus forcing unknown words to be decoded as close alternatives, improves answer accuracy. The methods are tested on synthetic TTS audio and on real spoken Quizbowl and Jeopardy! questions, providing a practical path for spoken virtual assistants.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human Quizbowl evaluation may be contaminated by training overlap: the paper never states the 50 human questions are held out, clean End accuracy is 1.000 (vs 0.540 synthetic), and the human sample is too small to support the human-data claim.","rationale":"I take the paper's central claim to be empirical: confidence features and forced decoding improve noisy-input QA, and the improvement transfers to real spoken questions. The synthetic half of the evidence is substantial: a 500k-sentence corpus, public code, and consistent relative gains (e.g., FD+Conf vs DAN on synthetic Quizbowl Start 0.041 vs 0.035, End 0.371 vs 0.335, Jeopardy 0.109 vs 0.097). I do not see an internal inconsistency there and I would not object to the synthetic claim on consensus grounds. The human half is the weak point. The paper itself limits the human evaluation to 50 Quizbowl recordings and one Jeopardy episode, and the observed improvements are two questions and one question. The clean-data 1.000 End accuracy on human Quizbowl, together with the absence of any holdout statement, makes train/test overlap a real correctness risk rather than a stylistic concern. This is the kind of missing support that should be flagged explicitly. If the overlap check comes back negative and a bootstrap CI excludes zero, the human claim would be supported and my concern would not land. Because the reader's verdict is already CONDITIONAL, I do not propose changing it; but I identify a different load-bearing condition than the reader's TTS/ASR distributional similarity assumption. This is a partial rather than full agreement.","tokens_in":8396,"tokens_out":15104,"duration_ms":162452,"concrete_test":"Using the released data, normalize and search the five human-recorded Quizbowl source questions (and their ASR transcripts) for exact or near-duplicate matches (e.g., high BLEU/Jaccard) against the 96k-question synthetic training corpus; if any overlap is found, the human Quizbowl rows of Table 2 are invalid. If no overlap is found, run a paired bootstrap over the 50 human Quizbowl questions comparing FD+Conf versus DAN and report the 95% confidence interval; an interval containing 0 would mean the claimed human gain is not statistically supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is explicitly about spoken Quizbowl and Jeopardy! questions, so the human rows of Table 2 are load-bearing. Those rows rest on 50 Quizbowl questions (five source questions x ten speakers) and one manually parsed Jeopardy episode. The paper never states that the five Quizbowl source questions are disjoint from the 96k-question corpus used to generate the synthetic training set, and Table 2 contains a red flag: on clean text, the DAN and IR baselines achieve 1.000 accuracy on human Quizbowl at End, while the same baselines reach only 0.540/0.544 on the synthetic Quizbowl End set. If the five human questions (or near-duplicates) appear in training, the human Quizbowl rows—including the headline FD+Conf improvement from 0.120 to 0.160 at Start—measure retrieval of training text rather than transfer to unseen spoken questions. The Jeopardy human evidence (one episode, a one-question apparent improvement) is too small to carry the human claim by itself, and no confidence intervals or significance tests accompany any human row. The Appendix A BLEU/WER overlap check addresses synthetic-vs-human error distributions but does not address sample size or train/test disjointness. The synthetic results are credible; the human-data component of the conclusion is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the problem of question answering (QA) systems receiving noisy inputs from automatic speech recognition (ASR). The authors construct a large synthetic corpus by passing roughly 96,000 Quizbowl questions and over 200,000 Jeopardy questions through Google Text-to-Speech and then through a Kaldi ASR system, producing noisy transcriptions with word-level confidence scores. They propose two mitigations: forced decoding, which removes the <unk> token from the ASR finite state transducer so that every word is decoded as a real word, and a confidence-augmented Deep Averaging Network (DAN) that learns to reweight word embeddings by a learned function of ASR confidence. They evaluate these methods against IR and DAN baselines on synthetic test sets and on small human speech corpora: 50 recorded Quizbowl questions (five unique questions across ten speakers) and one manually parsed Jeopardy episode. The paper reports that the combination of forced decoding and confidence features improves accuracy in several conditions, particularly for first-sentence Quizbowl accuracy, and claims that these methods improve QA accuracy on Quizbowl and Jeopardy questions.","tokens_in":8676,"tokens_out":6896,"duration_ms":70436,"significance":"If the results hold, the paper offers a practical, low-cost recipe for making QA systems more robust to ASR noise using only off-the-shelf TTS and ASR, which is relevant for virtual assistants and spoken trivia systems. The authors release code and data, and the paper includes negative results from noisy-channel and lattice-RNN attempts, which is a transparency strength. The synthetic corpus itself may be a useful resource. However, the central human-transfer claim is currently supported by a very small human evaluation and the paper does not establish that the human test questions are disjoint from the training corpus; in addition, some headline claims are not consistently supported by the tabulated numbers. These issues are fixable but require substantive revision.","major_comments":[{"comment":"The human Quizbowl evaluation does not establish transfer to unseen spoken questions. The paper never states that the five source questions used in the human recordings are disjoint from the 96,000-question Quizbowl corpus used to generate the synthetic training set, and the clean-text rows of Table 2 show 1.000 accuracy on the human Quizbowl End set versus 0.540 on the synthetic End set, which is consistent with training overlap. The human Jeopardy evidence is one manually parsed episode, so the reported 0.033 versus 0.017 for FD+Conf versus DAN corresponds to about one question, and no confidence intervals or significance tests are reported for any human row. Because the conclusion claims improvement on Quizbowl and Jeopardy questions without restricting to synthetic data, these rows are load-bearing; the authors should either confirm disjointness and add appropriate statistical measures or restrict the human-data claim.","section":"§4.1, Table 2"},{"comment":"The unqualified claim that confidence integration and forced decoding improve the accuracy of QA is not consistently supported by Table 2. For example, the Confidence model alone underperforms the DAN baseline on synthetic Jeopardy (0.095 vs. 0.097), forced decoding alone underperforms the DAN at synthetic Quizbowl Start (0.032 vs. 0.035), and several human rows are ties (e.g., Quizbowl Human End 0.440 for DAN, FD, and FD+Conf). The Table 2 caption's statement that combining the two methods leads to a further joint improvement is also contradicted by the synthetic Quizbowl End row, where Confidence alone (0.374) is higher than FD+Conf (0.371). The claims should be qualified to the specific conditions (e.g., first-sentence accuracy, particular datasets) where the improvements actually hold.","section":"Abstract and §5 vs. Table 2"},{"comment":"The validation of synthetic-to-human transfer is weaker than claimed. Figures 2 and 3 provide only a visual BLEU/WER distributional overlap on the same 50 human Quizbowl questions and one Jeopardy episode, with no statistical comparison (e.g., a two-sample test or effect size) and no direct evaluation of whether task accuracy on synthetic data predicts accuracy on human data under matched methods. The sentence in Appendix A that 'our results are comparable in accuracy to our synthetic data' is not backed by any quantitative comparison in the paper; the human rows of Table 2 are not directly compared against the synthetic rows with the same method and a test of difference. At minimum, the authors should report the sample size, the test statistic, and a confidence interval for the human rows, and temper the transfer conclusion accordingly.","section":"Appendix A, §4.1"}],"minor_comments":[{"comment":"The phrase 'it easier and cheaper' is missing 'is'; it should read 'it is easier and cheaper'.","section":"§1"},{"comment":"There is a doubled word in 'knows when when a word is not recognized'; one 'when' should be removed.","section":"§3.2"},{"comment":"'commerical' should be 'commercial'.","section":"§3.2"},{"comment":"The phrase 'cannot not reconstruct alternate paths' contains a double negative; it should be 'cannot reconstruct alternate paths'.","section":"§B.2"},{"comment":"The notation r** is not defined; the paper should introduce it explicitly or use a cleaner symbol such as r_conf.","section":"§3.3, Eq. (3)"},{"comment":"There is a formatting artifact 'textscir' in the sentence about IR methods; it should read 'IR'.","section":"§4"},{"comment":"The figures are described as color-coded by speaker, but the text-only rendering does not make the distinction clear; please ensure the figures are legible in grayscale and provide a colorblind-safe palette.","section":"Figures 2 and 3"},{"comment":"The caption and body would benefit from stating the number of test instances for each condition (especially human rows) so that the reader can judge the resolution of the reported differences.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is from an established group and includes open code/data and explicit negative results, which is commendable. The main concern is whether the human-data conclusions overreach; if the authors can confirm disjointness of the human test questions and add basic statistics, this could be a solid contribution. I would not reject based on the synthetic results, but the manuscript needs a careful revision of its claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper contributes a large synthetic TTS-to-ASR corpus for quizbowl QA and a clean forced-decoding trick that converts <unk> tokens into something a downstream model can use. The synthetic experiments support the central claim. The human-data part of the conclusion, however, is not established, and the stress-test note correctly identifies the gap: the paper never states that the five Quizbowl competition questions are disjoint from the 96k questions used for training, and clean human End accuracy of 1.000 (vs 0.540 synthetic) makes training overlap a real possibility. That needs to be resolved before the human rows of Table 2 are trusted.\n\nWhat's genuinely new? The synthetic corpus—over 500k TTS-generated noisy sentences—is a useful resource for the spoken QA setting. Using a learned scalar transform of word-level ASR confidences to weight DAN embeddings is a sensible adaptation of known confidence methods in MT, and forcing all <unk> to be decoded by removing the unknown token from the FST is simple and apparently effective. The paper also reports negative results (noisy channel expansion, lattice RNNs), which is honest, and the code is public.\n\nThe soft spots are real. The human evaluation is tiny: 50 Quizbowl utterances (five source questions times ten speakers) and one Jeopardy episode. There are no confidence intervals or significance tests anywhere, and with 50/61 samples, three-point differences are within noise. The Appendix A BLEU/WER overlap check is a distributional argument, not a substitute for sample size. The headline gains are modest: on synthetic Quizbowl End, FD+Conf moves from 0.335 (DAN baseline) to 0.371; on Jeopardy, 0.097 to 0.109. The conclusion's wording—'improves QA accuracy on Quizbowl and Jeopardy! questions'—is broader than the evidence, especially for human data.\n\nWho is this for? People building spoken QA systems or studying robustness to ASR noise. It deserves a serious referee: the synthetic result and dataset are credible, and the forced decoding idea is worth engaging. But a revision must clarify the disjointness of the human questions, report per-speaker variance, and either add significance testing or soften the human-transfer claim. I'd bring it to our reading group as a case study in how synthetic-to-real transfer claims can get ahead of the data.","headline":"The synthetic corpus and forced-decoding trick are real contributions, but the human-transfer claim rests on a tiny sample with a possible train/test overlap that the paper never rules out.","tokens_in":9208,"tokens_out":5044,"would_cite":true,"duration_ms":49813,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Spoken-question answering improves when the model reads ASR confidence and forces a guess instead of an unknown token.","keywords":["question answering","automatic speech recognition","ASR confidence","forced decoding","spoken question answering","deep averaging network","synthetic training data","noisy text"],"falsifier":"Record a larger set of human speakers reading the same Quizbowl and Jeopardy! questions, compare per-word ASR error types (substitutions, deletions, insertions) between the synthetic and human audio, and measure the forced-decoding-plus-confidence model's accuracy on both: if human speech produces systematically different error types or the accuracy gain disappears on the human set, the synthetic noise proxy is not reliable.","tokens_in":8212,"feed_emoji":"🎤","tokens_out":9940,"duration_ms":96325,"temperature":0.7,"pith_summary":"The paper addresses what happens when a question-answering system receives spoken questions through an automatic speech recognizer, which corrupts precisely the low-frequency words that carry the answer. Because large human speech corpora for QA are impractical to record, the authors synthesize over 500,000 noisy sentences by reading Quizbowl and Jeopardy! questions with text-to-speech and decoding them with an ASR system. They claim that a neural model can recover part of the accuracy lost to noise by weighting word embeddings with the recognizer's per-word confidence and by forcing the recognizer to output the closest known word instead of a generic unknown token. On human-recorded questions, the combination improves accuracy over the noisy-text neural baseline at the sentence level, though a retrieval baseline stays stronger on full multi-sentence Quizbowl questions.","feed_headline":"ASR confidence and forced decoding lift spoken-question QA accuracy","feed_subtitle":"Trains on synthetic speech, then recovers lost accuracy by weighting ASR confidence and guessing at unknown words.","key_machinery":"The first mechanism is forced decoding: the paper removes the unknown-token symbol from the ASR system's finite state transducer, so the recognizer must emit the acoustically closest known word instead of <unk>, converting a useless marker into an embeddable (if wrong) word. The second is a confidence-augmented Deep Averaging Network, which replaces the plain average of word embeddings with a confidence-weighted average $r^{**} = \\frac{1}{N}\\sum_i E[w_i]\\, f(c_i)$, where $f$ is a learned affine function $f(c) = W^{(c)}c + b^{(c)}$ of the recognizer's per-word confidence $c_i$; the resulting vector feeds the standard hidden layers and softmax classifier. Together they let the downstream model treat uncertainty as a feature rather than a failure.","core_discovery":"The central claim, stated in the paper's terms, is that a neural question-answering model which uses the ASR system's word-level confidence outputs and systematically forces decoding of words rather than unknowns improves QA accuracy on Quizbowl and Jeopardy! questions over a neural baseline trained and evaluated on the same noisy text. In the reported results, confidence weighting and forced decoding each help in some settings, and the combined model reaches the highest neural accuracy on the sentence-level and short-question settings; a strong retrieval baseline remains more accurate on full multi-sentence Quizbowl questions. The paper also reports that models trained on a synthetic corpus of text-to-speech audio transferred to human-recorded questions, based on BLEU and WER distribution overlap on a small human sample.","pith_inferences":["A natural next experiment would replace the single affine confidence function $f(c)$ with a small neural network per word, or feed phone-level confidences; the paper's ablation only shows that one learned affine transform helps, not that it is optimal.","The forced-decoding trick should transfer to recognizers that output a special token instead of <unk>; any system that replaces unknown words with the nearest known alternative and weights by confidence could see similar gains on other ASR-downstream tasks such as semantic parsing.","The Appendix's human-sample check uses only 50 Quizbowl questions and one Jeopardy episode, so the synthetic-to-human transfer claim would be much stronger if re-run on a larger corpus; the paper's own numbers suggest human WER is higher, so real-world gains could be smaller.","One could test whether the improvements are driven by high-confidence words being up-weighted or low-confidence words being down-weighted by ablating the learned $f$: clamping $f$ to the identity and comparing against the affine version would isolate the mechanism."],"forward_implications":["If the gains transfer, any embedding-averaging or bag-of-words downstream model can cheaply consume ASR uncertainty by adding a learned confidence scalar and forcing the recognizer to guess at unknown words, without retraining the ASR system.","The synthetic TTS-to-ASR pipeline supplies a scalable recipe for training supervised models for spoken input on tasks where no large human audio corpus exists.","The paper's results imply that for one-sentence or first-sentence questions, neural models that use confidence and forced decoding can outperform an IR baseline, so voice assistants handling short queries may benefit most directly.","Because the combination helps at the first-sentence level of Quizbowl, it could give a computer buzzer-accuracy an edge in the early, hard portion of a question, where humans often wait for easier clues."],"supporting_citations":[{"why":"Supplies the Deep Averaging Network architecture that the paper modifies with confidence weighting and evaluates as the neural baseline.","marker":"[18]"},{"why":"Describes the open-source speech recognition toolkit whose finite state transducer the paper edits to force decoding of unknown words.","marker":"[17]"},{"why":"Shows how ASR confidence and lattice information improve neural sequence models under uncertain input, the approach the paper adapts to question answering.","marker":"[6]"},{"why":"Provides the neural chain model used to decode the synthetic speech in the data-generation pipeline.","marker":"[7]"},{"why":"Provides the acoustic training data for the speech recognizer used throughout.","marker":"[8]"},{"why":"Source of the Quizbowl questions used to build the large synthetic noisy corpus.","marker":"[2]"},{"why":"Source of the Jeopardy! question set used as the second task corpus.","marker":"[14]"},{"why":"Defines the BM25 retrieval scoring used by the strong IR baseline that the paper compares against.","marker":"[16]"}],"fun_headline_variants":["Confidence and forced decoding tame ASR noise for QA","ASR noise mitigation boosts neural QA on Quizbowl and Jeopardy","Confidence weighting and forced decoding improve spoken QA accuracy","Neural QA recovers lost accuracy from ASR noise via confidence and decoding","ASR noise confounds QA yet confidence and forced decoding help"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ASR errors on synthetic text-to-speech audio match ASR errors on real human speech well enough that models trained on the synthetic corpus transfer to human-recorded questions, a premise checked only indirectly through distribution overlap on a small human sample.","fun_headline_variants_meta":{"raw":{"variants":["Confidence and forced decoding tame ASR noise for QA","ASR noise mitigation boosts neural QA on Quizbowl and Jeopardy","Confidence weighting and forced decoding improve spoken QA accuracy","Neural QA recovers lost accuracy from ASR noise via confidence and decoding","ASR noise confounds QA yet confidence and forced decoding help"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000653,"raw_usage":{"total_tokens":2915,"prompt_tokens":789,"completion_tokens":2126,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":405,"completion_tokens_details":{"reasoning_tokens":2037}},"tokens_in":405,"tokens_out":2126,"duration_ms":16174,"temperature":1.0,"reasoning_tokens":2037,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:29:13.225810+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record a larger set of human speakers reading the same Quizbowl and Jeopardy! questions, compare per-word ASR error types (substitutions, deletions, insertions) between the synthetic and human audio, and measure the forced-decoding-plus-confidence model's accuracy on both: if human speech produces systematically different error types or the accuracy gain disappears on the human set, the synthetic noise proxy is not reliable.","supporting_citations":[{"cited_title":"Odsqa: Open-domain spoken question answering dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the Deep Averaging Network architecture that the paper modifies with confidence weighting and evaluates as the neural baseline."},{"cited_title":"Exploring speech enhancement with generative adversarial networks for ro- bust speech recognition,","cited_arxiv_id":null,"evidence_quote":"Describes the open-source speech recognition toolkit whose finite state transducer the paper edits to force decoding of unknown words."},{"cited_title":"The views expressed in this paper are our own","cited_arxiv_id":null,"evidence_quote":"Shows how ASR confidence and lattice information improve neural sequence models under uncertain input, the approach the paper adapts to question answering."},{"cited_title":"Build Watson: an overview of DeepQA for the Jeopardy! challenge,","cited_arxiv_id":null,"evidence_quote":"Provides the neural chain model used to decode the synthetic speech in the data-generation pipeline."},{"cited_title":"Boyd-Graber, S","cited_arxiv_id":null,"evidence_quote":"Provides the acoustic training data for the speech recognizer used throughout."},{"cited_title":"cyclohexane","cited_arxiv_id":null,"evidence_quote":"Source of the Quizbowl questions used to build the large synthetic noisy corpus."},{"cited_title":"The fisher corpus: a resource for the next generations of speech-to-text,","cited_arxiv_id":null,"evidence_quote":"Source of the Jeopardy! question set used as the second task corpus."},{"cited_title":"Synthetic and natural noise both break neural machine translation,","cited_arxiv_id":null,"evidence_quote":"Defines the BM25 retrieval scoring used by the strong IR baseline that the paper compares against."}],"review_version":1}