{"id":"a626668c-718e-45a5-88b9-2d33882e82b0","arxiv_id":"2412.16500","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SpeechRAG retrieves audio passages directly with a speech adapter feeding a frozen text retriever, matching ground-truth text retrieval and beating ASR cascades on high-WER speech.","lead":"Researchers built a speech retrieval system that searches audio passages directly from text questions, skipping speech to text transcription. It matches text-only retrieval on two spoken question answering datasets and beats transcription-based pipelines when transcriptions are noisy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The retrieval-parity claim rests on a train/eval split that the paper never states; without a passage-disjoint split, Table II's recall may reflect memorization of training passages rather than generalizable direct speech retrieval.","rationale":"The reader's weakest assumption correctly identifies the missing train/test split as the load-bearing weakness, and my independent reading of Sections IV-A, IV-B, and Eq. (1) leads to the same conclusion. The reported retrieval results are internally consistent and the noise-robustness experiment in Fig. 2 is a useful additional observation, but neither compensates for the absence of a stated passage-disjoint evaluation. Because the paper could be conditional rather than wrong, and because the authors may have used a split that they simply omitted to describe, the appropriate verdict remains CONDITIONAL. My read does not move the verdict, so I mark it UNCHANGED. The concrete test of retraining on a passage-disjoint split and recomputing Recall@5 on held-out passages would settle whether the parity with text retrieval is a genuine architectural property or a memorization effect.","tokens_in":8425,"tokens_out":2963,"duration_ms":49403,"concrete_test":"Retrain the speech adapter on an explicitly passage-disjoint split of SpokenSQuAD (e.g., using only training-passage audio/transcript pairs for the distillation loss in Eq. (1)), then recompute Recall@5 on held-out SpokenSQuAD passages and compare against the GT Text Baseline in Table II. If the held-out SpeechRAG Recall@5 falls more than 0.01-0.02 below the GT text baseline, the reported parity is an in-domain artifact rather than evidence of generalizable direct speech retrieval.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that direct speech retrieval 'does not degrade over the text-based baseline' depends on the speech adapter learning a general alignment between speech and the frozen text retriever's embedding space. The adapter is trained with the cosine distillation loss in Eq. (1), using ground-truth transcript embeddings of the same datasets on which retrieval is evaluated. Section IV-A describes SpokenSQuAD and VoxPopuli but gives no train/validation/test partition; Section IV-B reports training for 20 epochs with early stopping on validation loss and a patience of 3, but never states whether the passages used for evaluation were seen during adapter training. If the evaluation passages overlap with the training passages, the reported SpeechRAG Recall@5 values (0.9702 on SpokenSQuAD, 0.9952 on VoxPopuli) can be achieved by overfitting the adapter to reproduce the text embeddings of those exact passages, rather than by a robust text-to-speech alignment that generalizes to unseen audio. This is especially plausible because VoxPopuli queries are generated from the same utterances that are indexed, so each query has a trivially aligned target passage. The concern is not that the numbers are fabricated; it is that the paper omits the experimental control needed to distinguish genuine cross-modal generalization from in-domain memorization. Without a stated passage-disjoint split, the headline parity with the ground-truth text baseline is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SpeechRAG, a framework for open-domain question answering over spoken passages that bypasses ASR at both retrieval and generation time. A HuBERT encoder plus a lightweight adapter is trained with cosine distillation (Eq. 1) to map speech into the embedding space of a frozen E5-Mistral text retriever; retrieval is done by cosine similarity between text queries and audio passage embeddings. Generation is performed by a frozen speech language model (Qwen-Audio-Chat) conditioned on retrieved audio passages. On SpokenSQuAD and a self-built VoxPopuli QA benchmark, the speech retriever achieves Recall@5 close to the ground-truth text baseline (0.9702 vs 0.9707 and 0.9952 vs 0.9942) and outperforms high-WER cascaded ASR baselines. The generation experiments show SpeechRAG outperforms high-WER cascades but underperforms low-WER cascades and the ground-truth text baseline, a result the paper reports honestly.","tokens_in":8739,"tokens_out":2655,"duration_ms":25164,"significance":"If the retrieval-parity claim is robust, the paper makes a useful contribution: it shows that a lightweight speech adapter plus a frozen LLM-based text retriever can avoid ASR error propagation in spoken retrieval, and it provides a concrete recipe that avoids expensive cross-modal contrastive pretraining. The noise-robustness experiment is a valuable additional result, and the paper is transparent about the generation gap against low-WER cascades. The main limitation is evidential: because the training/evaluation split is not specified, the core parity result may reflect in-domain distillation rather than generalizable cross-modal alignment, and the reported differences are presented without error bars or significance checks. The architecture and the honest reporting of the generation results are strengths, but the central retrieval claim needs stronger experimental support.","major_comments":[{"comment":"The paper never states the train/validation/test partition for SpokenSQuAD or VoxPopuli. The adapter is trained with the cosine distillation loss in Eq. (1) using ground-truth transcript embeddings of passages from the same two datasets on which retrieval is evaluated, and Section IV-B mentions early stopping on validation loss but does not say whether the evaluation passages were excluded from training. If the evaluation passages overlap with the training set, the Recall@5 values in Table II (0.9702 on SpokenSQuAD, 0.9952 on VoxPopuli) can be achieved by memorizing the text embeddings of those exact passages, and the headline claim of parity with ground-truth text retrieval is not established as a generalization result. Please specify the split, ensure passage-disjoint training/evaluation, and report results on a held-out split.","section":"Section IV-A and IV-B"},{"comment":"The retrieval results are reported as single point estimates with no error bars, confidence intervals, or significance tests. The differences at issue are very small: the Speech Retriever is 0.0005 below the GT text baseline on SpokenSQuAD Recall@5 and 0.0010 above it on VoxPopuli Recall@5, while both methods are within 0.001 on most rows. Without knowing the number of queries, the variance across random seeds or dataset subsamples, and the significance of these differences, the claim that direct speech retrieval 'does not degrade' over the text baseline is not statistically supported. Please report uncertainty estimates and, ideally, significance tests for the key comparisons.","section":"Section V-A, Table II"},{"comment":"The generation experiments support the claim that SpeechRAG beats high-WER cascades, but the paper should also state explicitly in the abstract and conclusions that SpeechRAG underperforms low-WER cascades by large margins (e.g., on SpokenSQuAD, EM 0.3522 vs 0.5019 for Low WER fully-cascaded; on VoxPopuli, EM 0.8045 vs 0.7473 for Low WER fully-cascaded is closer, but LLM Correctness is lower). The current abstract says the framework 'outperforms cascaded text-based models when there is high WER,' which is accurate, but the broader framing should not imply that the proposed system is generally superior to cascades.","section":"Section V-C and Table III"}],"minor_comments":[{"comment":"The noise robustness experiment in Fig. 2 is described as comparing 'text-based' versus the end-to-end retriever, but the text says the comparison is with 'the cascaded baseline that uses the same speech encoder as our end-to-end retriever.' Please clarify the exact baselines in the figure caption and text.","section":"Section V-B"},{"comment":"The description of the Low WER cascaded baseline is vague: it says 'the case uses the same audio encoder used for the speech adapter,' but it is not clear which ASR model or decoding strategy produces the reported ~17-20% WER. Please specify the ASR model and how its outputs were obtained.","section":"Section IV-D"},{"comment":"The training configuration reports learning rate, batch size, gradient accumulation, and early stopping, but the number of training examples and the number of validation queries are not given. Please add dataset sizes and the number of query-passage pairs used for evaluation.","section":"Section IV-B"},{"comment":"The sentence 'Table 3 shows how SpeechRAG avoids the corruption of context information' should reference Table III consistently (the text uses 'Table 3' while the table is numbered III).","section":"Section V-C"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be an ICASSP-style submission, and the experimental reporting falls short of journal standards. The most serious issue is the missing train/eval split description; if the authors can clarify that the evaluation passages are disjoint from training and provide error bars, the retrieval-parity claim may be salvageable. I would recommend requesting a major revision rather than rejection, because the reported generation comparison is honest and the architecture is simple and likely reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the thing. The core trick is genuinely neat: take a frozen LLM-based text retriever, train a lightweight adapter that maps HuBERT speech features into that same embedding space via cosine distillation to ground-truth transcript embeddings, and then retrieve audio with text queries. That avoids training a cross-modal retriever from scratch and avoids contrastive learning. It's a real recipe, and the retrieval numbers on SpokenSQuAD and VoxPopuli are strong: roughly parity with ground-truth text retrieval and clearly better than high-WER cascades. The noise-robustness experiment is a nice bonus. The generation section is also honest: SpeechRAG beats the high-WER cascades, loses to low-WER ones, and the authors say why (long audio in SpokenSQuAD hurts the SLM).\n\nThe problem is the experimental control. Section IV-B never states a train/eval split. The adapter is trained on ground-truth transcripts from the same two datasets used for evaluation, and the retrieval results are reported on those datasets without saying whether the evaluation passages were held out. If the adapter saw those exact passages during training, then the near-perfect parity with text retrieval could be memorization of passage embeddings, not a generalizable speech-to-text alignment. The VoxPopuli case makes this worse: queries are mined from the same utterances that are indexed, so each query has a trivially aligned passage. Without a passage-disjoint split, the headline claim 'direct speech retrieval does not degrade over the text-based baseline' is not established. This is a load-bearing omission, not a cosmetic one. The fix is simple: state the split, or run a held-out passage experiment.\n\nThere are also no error bars or significance tests anywhere, which matters because the reported differences between SpeechRAG and the GT text baseline are small (a few tenths of a percent). That's a minor issue compared to the split, but worth asking about.\n\nMy take: the idea is worth taking seriously, and the paper is otherwise well written and properly cites prior work like SpeechDPR and ReSLM. It deserves peer review, and a good referee should push for a held-out split and error bars before trusting the parity claim. The generation story is already modest and credible. This is a solid ICASSP-level contribution conditional on that experimental control.","headline":"Promising ASR-free retrieval recipe, but the missing train/eval split undermines the headline parity claim.","tokens_in":9251,"tokens_out":2688,"would_cite":true,"duration_ms":24026,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SpeechRAG claims that a speech adapter aligned to a frozen LLM text retriever can retrieve audio passages from text queries as accurately as ground-truth transcripts, while avoiding ASR error propagation.","keywords":["speech retrieval-augmented generation","spoken content retrieval","cross-modal retrieval","multimodal retrieval","open question answering","audio language model","speech language model"],"falsifier":"Train the speech adapter on one spoken QA dataset, such as SpokenSQuAD, and evaluate it zero-shot on another, such as VoxPopuli; if Recall@5 falls far below the ground-truth text baseline on the unseen dataset, the method's parity is an artifact of training on the evaluation distribution rather than a general alignment. A second check is to hold out a random split of one dataset and compare the speech adapter against the text retriever on the same split, since the paper does not state whether such a split was used.","tokens_in":8283,"feed_emoji":"🎙️","tokens_out":8584,"duration_ms":64452,"temperature":0.7,"pith_summary":"SpeechRAG aims to show that question answering over spoken audio does not need a speech-to-text step. The paper trains a lightweight speech adapter that projects audio into the embedding space of a frozen, LLM-based text retriever, so text queries can retrieve audio passages directly. In experiments on two spoken question-answering datasets, this direct retriever matches the ground-truth transcript baseline (Recall@5 of 0.9702 vs 0.9707 on SpokenSQuAD, 0.9952 vs 0.9942 on VoxPopuli) and clearly beats cascaded ASR-based retrieval when transcripts are noisy. The same idea carries through generation: a pre-trained speech language model conditioned on retrieved audio, not transcripts, outperforms cascaded text-only generation when ASR word error rate is high. The contribution is a way to remove ASR error propagation from both retrieval and generation without training a cross-modal retriever from scratch.","feed_headline":"Without ASR, speech retrieval matches text retrieval","feed_subtitle":"A speech adapter aligns audio with a frozen text retriever, matching text baselines and beating noisy ASR pipelines at top-5 retrieval.","key_machinery":"The load-bearing object is the speech adapter: a trainable projection inserted between a pre-trained speech encoder (HuBERT-large) and a frozen LLM-based text retriever (E5-Mistral-7B-Instruct). Speech is encoded into frame-level features, downsampled by average pooling, and projected into the retriever's token-embedding dimension; the same frozen retriever then processes both text queries and speech passages. The adapter is trained by cosine-embedding distillation, $L = 1 - \\cos(\\mathbf{e}_s, \\mathbf{e}_t)$, pulling the speech embedding $\\mathbf{e}_s$ toward the text embedding $\\mathbf{e}_t$ of the passage's ground-truth transcript. This alignment lets a single frozen text retriever do cross-modal search without contrastive learning, and the same frozen retriever's capacity is what makes parity with text retrieval plausible.","core_discovery":"The paper's central claim is that direct speech retrieval does not degrade relative to text retrieval on ground-truth transcripts, and that it outperforms cascaded systems that rely on ASR. The evidence is the retrieval table: on SpokenSQuAD the speech retriever reaches Recall@5 of 0.9702 against 0.9707 for the ground-truth text baseline and 0.8768 for the high-WER cascade; on VoxPopuli it reaches 0.9952 against 0.9942 and 0.7106 respectively. The paper also claims that conditioning a speech language model on the retrieved audio, rather than on ASR transcripts, yields better generated answers than cascaded text generators when the transcripts have high word error rate, with exact-match scores of 0.3522 vs 0.2684 on SpokenSQuAD and 0.8045 vs 0.4511 on VoxPopuli. The reason this is significant is that ASR errors, especially on named entities, are avoided entirely in the retrieval-and-generation path.","pith_inferences":["Editorial inference: the absence of a stated train/test split means the headline parity could be in-domain distillation; a held-out-corpus test would tell whether the adapter learns general text-to-speech alignment.","Editorial inference: the same frozen-retriever-plus-adapter pattern could be applied to other modalities, such as image or video, wherever paired text descriptions and a pre-trained encoder exist.","Editorial inference: if long-context audio is what limits the speech language model, retrieving shorter passages or adapting the SLM to multi-passage audio would likely narrow the low-WER generation gap."],"forward_implications":["Retrieval over spoken archives can be built without an ASR component, eliminating transcription-error propagation at the indexing and search stages.","The speech retriever's Recall@k is on par with ground-truth text retrieval, so for retrieval purposes audio can substitute for clean transcripts.","Under added Gaussian noise, the direct speech retriever holds up better than a cascade using the same speech encoder, except at very high noise on the TTS-based SpokenSQuAD set.","In high-WER settings, audio-conditioned generation with a speech language model answers more accurately than text-generation pipelines that read noisy transcripts.","The gap against low-WER cascades in generation remains, with SpeechRAG performing worse than both fully- and semi-cascaded systems when transcripts are clean."],"supporting_citations":[{"why":"Defines retrieval-augmented generation, the text framework SpeechRAG extends to speech.","marker":"[1]"},{"why":"The closest end-to-end spoken passage retrieval system, motivating the teacher-distillation design.","marker":"[21]"},{"why":"Provides the speech-adapter architecture and training recipe the paper adapts.","marker":"[23]"},{"why":"Supplies the pre-trained HuBERT speech encoder used to obtain frame-level speech features.","marker":"[24]"},{"why":"The frozen LLM-based text retriever whose embedding space the adapter learns to match.","marker":"[27]"},{"why":"The speech language model used as the audio-conditioned generator.","marker":"[28]"},{"why":"One of the two evaluation datasets, a spoken version of SQuAD.","marker":"[29]"},{"why":"The other evaluation dataset, European Parliament speech with mined query-answer pairs.","marker":"[30]"}],"fun_headline_variants":["SpeechRAG: no ASR, speech retrieval matches text","Direct speech retrieval beats ASR cascades in QA","Skip ASR: audio-only retrieval rivals text baseline","SpeechRAG: tune speech encoder, keep text retriever","Audio-first RAG outperforms noisy transcription pipelines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the speech adapter is trained on ground-truth transcripts from the same datasets it is later evaluated on, so the parity with text retrieval may be an in-domain distillation effect rather than evidence that spoken passages in general can be embedded into the text retriever's space.","fun_headline_variants_meta":{"raw":{"variants":["SpeechRAG: no ASR, speech retrieval matches text","Direct speech retrieval beats ASR cascades in QA","Skip ASR: audio-only retrieval rivals text baseline","SpeechRAG: tune speech encoder, keep text retriever","Audio-first RAG outperforms noisy transcription pipelines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00015,"raw_usage":{"total_tokens":1213,"prompt_tokens":978,"completion_tokens":235,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":155}},"tokens_in":594,"tokens_out":235,"duration_ms":2733,"temperature":1.0,"reasoning_tokens":155,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:30:19.765206+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the speech adapter on one spoken QA dataset, such as SpokenSQuAD, and evaluate it zero-shot on another, such as VoxPopuli; if Recall@5 falls far below the ground-truth text baseline on the unseen dataset, the method's parity is an artifact of training on the evaluation distribution rather than a general alignment. A second check is to hold out a random split of one dataset and compare the speech adapter against the text retriever on the same split, since the paper does not state whether such a split was used.","supporting_citations":[{"cited_title":"Retrieval- augmented generation for knowledge-intensive nlp tasks,","cited_arxiv_id":null,"evidence_quote":"Defines retrieval-augmented generation, the text framework SpeechRAG extends to speech."},{"cited_title":"Speechdpr: End-to-end spoken passage retrieval for open-domain spoken question answering,","cited_arxiv_id":null,"evidence_quote":"The closest end-to-end spoken passage retrieval system, motivating the teacher-distillation design."},{"cited_title":"Speechverse: A large-scale generalizable audio language model,","cited_arxiv_id":null,"evidence_quote":"Provides the speech-adapter architecture and training recipe the paper adapts."},{"cited_title":"Hubert: Self- supervised speech representation learning by masked prediction of hidden units,","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained HuBERT speech encoder used to obtain frame-level speech features."},{"cited_title":"Improving text embeddings with large language models,","cited_arxiv_id":null,"evidence_quote":"The frozen LLM-based text retriever whose embedding space the adapter learns to match."},{"cited_title":"Spoken squad: A study of mitigating the impact of speech recognition errors on listening comprehension,","cited_arxiv_id":null,"evidence_quote":"One of the two evaluation datasets, a spoken version of SQuAD."},{"cited_title":"V oxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,","cited_arxiv_id":null,"evidence_quote":"The other evaluation dataset, European Parliament speech with mined query-answer pairs."}],"review_version":1}