{"id":"b3795d7f-0c0a-4bc7-acf5-0098814c39df","arxiv_id":"2412.12009","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A query-guided token pruning method improves speech-LLM accuracy on a new long-form audio benchmark while cutting computation.","lead":"This paper creates a benchmark of 1,012 near-90-second spoken conversations with questions, and proposes a way to discard irrelevant audio before feeding it to a speech AI model. The method uses the written question to find the relevant audio region, reporting accuracy gains of 29 to 47 percentage points over baselines while cutting computation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The main 'original model' baseline is a 30-second truncation, so the reported 29% gain may reflect access to full audio, not pruning quality.","rationale":"The paper's central claim is an empirical one about accuracy on SPIRAL. The reader's weakest_assumption identifies query-option leakage and open-ended generalization; that is a real limitation for the SIR task framing, but it does not invalidate the reported SPIRAL numbers because SPIRAL is multiple-choice by construction. The more immediate threat to the central numbers is the Original baseline: Section IV-A's 'Original' is not the unpruned model on the full audio, but a 30-second truncation. Since the method's first phase operates on the full audio, the 29% improvement over Original may be due to seeing content after the 30-second cut. This is not an internal inconsistency, but it is a missing control: without a full-context unpruned baseline, the quantitative headline is underdetermined. RAP/RAC comparisons show that semantic selection beats random selection from the full audio, so the method likely has real value; the concern is about the specific claim of improvement over the original model. A single full-length unpruned run would resolve this. The reader's conditional verdict already asks for this control, so my stress test does not change the verdict; it sharpens why condition (1) is load-bearing rather than cosmetic.","tokens_in":8719,"tokens_out":7383,"duration_ms":66705,"concrete_test":"Run Qwen2-Audio on all 1,012 full-length SPIRAL samples with no pruning or truncation (using an 80GB GPU, or a chunked-with-cross-attention implementation that reads all audio chunks), and compare accuracy, TFLOPS, prefill time, and memory against the 60.38% Original number. If full-context accuracy is at or above 89.23%, the improvement over Original is an artifact of the 30s baseline; if it is near or below 60.38%, the pruning claim is strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-A defines the Original baseline as 'full audio trimmed at 30 seconds (750 tokens)'. SPIRAL audio averages 87.89 seconds (Section II-C), and Qwen2-Audio is stated to be unable to process longer inputs on the available GPU. SpeechPrune, by contrast, prunes the full 90-second sequence: the first phase selects 750 tokens from the entire audio, and the second phase removes tokens to the target rate. Thus at 20% pruning the method feeds the model 600 tokens drawn from the full context, while 'Original' feeds 750 tokens from only the first ~30 seconds. The reported 89.23% vs. 60.38% gap therefore conflates two variables: pruning quality and audio coverage. The same issue affects the 'maintains performance at 80% pruning' claim (62.45% at 0.8 vs. 60.38% Original), since the reference is a truncated-input baseline. The RAP/RAC comparisons are less contaminated because those baselines also sample from the full audio, but they do not validate the headline 'over the original model' claim. Unless a full-length unpruned baseline is run, the central quantitative claim is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Speech Information Retrieval (SIR), a long-context task for speech large language models, and presents SPIRAL, a 1,012-sample benchmark of roughly 90-second spoken dialogues with multiple-choice questions. It then proposes SpeechPrune, a training-free two-phase token pruning method that first selects audio tokens by cosine similarity to the text query and then refines the selection using binarized first-layer attention weights. Experiments on Qwen2-Audio and DiVa report large accuracy gains over a 30-second-truncated baseline and over random pruning baselines, alongside substantial reductions in TFLOPS, prefill time, and activation memory.","tokens_in":8917,"tokens_out":4989,"duration_ms":46145,"significance":"If the central comparison is made fair, the paper would be a useful contribution: it identifies an under-explored long-context speech task, contributes an open benchmark, and proposes a parameter-free, plug-and-play pruning method that is evaluated on external benchmarks (DREAM-TTS, CN-College-Listen) and a second model (DiVa). The absence of fitted parameters and the use of external benchmarks are genuine strengths that reduce circularity concerns. However, the headline claim of '29% improvement over the original model' rests on a baseline that sees only the first 30 seconds of audio, while SpeechPrune sees the full audio, so the claim as stated is not yet established.","major_comments":[{"comment":"The 'Original' baseline is defined as 'full audio trimmed at 30 seconds (750 tokens).' Since SPIRAL audio averages 87.89 seconds, SpeechPrune at a 20% pruning rate feeds the model 600 tokens drawn from the full audio, whereas Original feeds 750 tokens from only the first 30 seconds. The reported 89.23% versus 60.38% therefore conflates pruning quality with full-context access. This is confirmed by the ablation in §IV-C: 'All Inputs' on SPIRAL-H achieves 43.6%, while Original achieves 0%. Please add an unpruned full-length baseline on SPIRAL (for instance, via chunking or a longer-context model) or explicitly reframe the claims as gains over a truncated baseline; as it stands, the headline '29% over the original model' is not established.","section":"§IV-A, Table I"},{"comment":"SPIRAL-H is defined as the 401 cases where the original Qwen-2 Audio model achieves 0% accuracy. Because that original model is the 30-second truncated baseline, SPIRAL-H is selected for failure under truncation, not for difficulty under full-input processing. This makes comparisons such as 81.64% versus 0% structurally favorable to SpeechPrune. Please define the hard subset using a full-input baseline, or report results on a random subset of SPIRAL, and include the full-input baseline on SPIRAL-H.","section":"§II-C and §IV-A"},{"comment":"First-phase pruning computes cosine similarity between audio tokens and T, which in SPIRAL includes the four multiple-choice answer options. The correct option's wording can therefore act as a direct lexical or semantic pointer to the answer-bearing audio segment. The claim that SIR 'can be easily generalized to open-ended questions' is not tested anywhere in the paper. Please run open-ended experiments or explicitly discuss this limitation and its potential effect on the method's locating power when answer-specific wording is absent.","section":"§III-B and §II-A"},{"comment":"For DREAM-TTS and CN-College-Listen, the setup for the 'Qwen-2 Audio' baseline is not specified. If those baselines also use 30-second truncation for samples exceeding 60 seconds, the external-benchmark evidence inherits the same confound as Table I. Please state explicitly whether the baseline uses full audio and how the model handles inputs longer than 30 seconds, or restrict the comparison to baselines with identical audio coverage.","section":"§IV-D, Table II"}],"minor_comments":[{"comment":"The phrases '29% higher accuracy' and 'up to 47%' describe absolute percentage-point differences (89.23% vs. 60.38% and 89.23% vs. 42.49%), not relative improvements. Please use 'percentage points' or report relative percentages to avoid ambiguity.","section":"Abstract and §I"},{"comment":"The frame size f is described as 'frame size per second' but its numerical value is never given. For Whisper-based encoders, 30 seconds corresponds to 750 tokens, so f would be 25; please state the value used and clarify that Eq. (5) first averages over text tokens and then sums over tokens within a frame.","section":"§III-B, Eq. (5)"},{"comment":"The 'All Inputs' accuracy of 43.6% on SPIRAL-H appears inconsistent with Original's 0% in Table I unless 'All Inputs' means full-length unpruned audio rather than the 30-second trimmed input. Please clarify this in both the text and the figure caption.","section":"§IV-C, Fig. 3"},{"comment":"There is a typo in 'hierarchial orgnization' in the Transcript Generation subsection; it should read 'hierarchical organization.'","section":"§II-B"},{"comment":"Table I reports point estimates without confidence intervals or significance tests. Given the sample sizes (1,012 and 401), adding error bars or a paired test would strengthen the claim that SpeechPrune 'maintains network performance even at a pruning level of 80%.'","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and the pruning method are potentially valuable, and the comparisons against random pruning over full audio are credible. The main barrier is the unfair 'Original' baseline; a full-length unpruned baseline is necessary before the headline claim can be accepted. The SPIRAL-H selection criterion should also be revisited. I would be supportive of a revised version that addresses these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know: this is a genuinely useful paper on a real problem, but the headline number overstates the method. The 'Original' baseline is a 30-second truncation, so the 29% gain on SPIRAL mixes access to full audio with pruning quality. Also, SPIRAL-H is selected by baseline failure, so the 0%→81% story is diagnostic, not a headline.\n\nWhat's actually new: the SIR task formulation, the SPIRAL benchmark (1,012 long-form samples with realistic TTS), and SpeechPrune, a training-free two-phase token pruning method using cosine similarity plus binarized attention. The adaptive frame allocation is a nice touch. The method is parameter-free, tested on two external benchmarks and a second model, and the gains there are consistent, though those baselines are under-specified. The efficiency numbers are concrete. It also ships an ablation on SPIRAL-H showing full-input accuracy of 43.6% vs 81.64% pruned, which shows pruning genuinely helps beyond just seeing all the audio, at least on that subset.\n\nSoft spots: the main comparison is apples-to-oranges. At 20% pruning, the method feeds 600 tokens drawn from ~90 seconds while 'Original' feeds 750 tokens from the first 30 seconds. The 89.23 vs 60.38 gap is not a pruning effect alone. The ablation helps but it's only on SPIRAL-H, so the full SPIRAL full-input baseline is still missing. The paper should either run that baseline or move the 'over the original model' claim to the external benchmarks where they can control for input length. Second, SPIRAL-H should be presented as a case study, not a headline subset. Third, the novelty claim should be checked against LongAudioBench and other pre-2024 long-form audio QA benchmarks; uncited is a real oversight. Fourth, the open-ended generalization is asserted without experiments. Also, code and dataset URLs need to be released for reproducibility.\n\nThe paper deserves a serious referee. The core direction is sound, the method is concrete, and the benchmark is likely to be used. But the authors need to rerun a full-length baseline and fix the framing before acceptance.","headline":"Useful, concrete pruning method for long-form speech QA, but the headline 29% gain is confounded by the truncated 'Original' baseline; needs a full-audio reference and toned-down SPIRAL-H framing.","tokens_in":9496,"tokens_out":2855,"would_cite":true,"duration_ms":25139,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SpeechPrune: pruning audio tokens by query similarity and attention lifts speech retrieval accuracy by 29 points and cuts compute by 70%.","keywords":["speech large language models","speech information retrieval","token pruning","long-context audio","SPIRAL benchmark","cross-modal similarity","attention approximation","training-free compression"],"falsifier":"Run SpeechPrune on SPIRAL-style questions where the correct answer is paraphrased rather than stated in the audio, or on open-ended questions with no answer options; if the 29-percentage-point gain at 20% pruning shrinks to parity with random pruning, the query-option overlap is what carries the method.","tokens_in":8485,"feed_emoji":"🎧","tokens_out":9423,"duration_ms":81153,"temperature":0.7,"pith_summary":"The paper proposes that long-form speech understanding fails partly because Speech LLMs drown in audio tokens, and that a training-free pruning method can fix both efficiency and accuracy. It defines Speech Information Retrieval (SIR), the task of finding one critical detail in roughly 90 seconds of spoken input, and contributes SPIRAL, a 1,012-sample multiple-choice benchmark built from synthesized lectures, meetings, and conversations. On this benchmark, SpeechPrune keeps audio tokens that resemble the query and that receive high binarized attention, raising accuracy by 29 percentage points over the unpruned model at 20% pruning and still matching it at 80% pruning while cutting FLOPs by 70%. The upshot is that off-the-shelf Speech LLMs can handle longer audio than their usual 30-second restriction suggests, if the context is pruned intelligently.","feed_headline":"Query-guided pruning lifts speech retrieval accuracy 29 points","feed_subtitle":"A training-free method keeps long-form accuracy intact at 80% pruning while cutting 70% of compute.","key_machinery":"The load-bearing mechanism is a two-phase, training-free token selector. Phase 1 derives a speech-text similarity matrix between audio-token embeddings $S$ and the text-query embeddings $T$ (excluding system tokens), groups the audio into one-second frames, and uses a softmax over mean frame scores to allocate each frame a budget of tokens; within each frame the top tokens by mean similarity to the query survive. Phase 2 takes the survivors, binarizes the first-layer query/key weights and token embeddings by sign, computes approximate attention scores $A = \\mathrm{softmax}(Q'K'^{\\top}/\\sqrt{d_k})$, and keeps the speech tokens with the largest mean attention. The two phases are complementary: the first finds query-relevant audio, the second keeps the internally important speech context, and both cost far less than a full forward pass because only one layer's binarized attention is estimated.","core_discovery":"SpeechPrune claims that token pruning is not a necessary evil for long audio but an accuracy booster. Because the answer to a speech-information-retrieval question lives in a small fraction of a ~90-second clip, keeping only the audio tokens most similar to the query and then re-ranking them by a binarized first-layer attention score removes the distracting majority. On SPIRAL, at 20% pruning the method raises Qwen2-Audio's accuracy from 60.38% to 89.23%, and from 0% to 81.64% on SPIRAL-H, a 401-sample subset where the original model fails entirely. At 80% pruning it keeps 62.45% accuracy, essentially the original model's 60.38%, while reducing FLOPs from 12.2 to 3.66 TFLOPS, prefill time from 779 to 278 ms, and activation storage from 0.19 to 0.04 GB. The same recipe improves accuracy on DREAM-TTS, CN-College-Listen, and the DiVA model, so the paper frames pruning as a general plug-in for off-the-shelf Speech LLMs.","pith_inferences":["A direct extension the paper does not run is open-ended SIR: SpeechPrune's Phase 1 uses the four answer options to locate the answer, so the reported gains likely depend on that multiple-choice scaffolding, and question-only queries would bound the method's real-world range.","The 80%-pruning result hints that the underlying problem is attention dilution rather than missing information; if so, other context-focusing strategies, such as two-pass retrieval with a cheap audio tagger, could approach the same gains.","The one-second frame size in Phase 1 is motivated by a claim about delta-band speech processing, and a sweep of frame sizes on SPIRAL would show whether that specific boundary is load-bearing or merely convenient."],"forward_implications":["At a 20% pruning rate, SpeechPrune raises SPIRAL accuracy from 60.38% to 89.23% and SPIRAL-H from 0% to 81.64%, outperforming random audio pruning and cropping.","At an 80% pruning rate, accuracy stays at 62.45% versus the original 60.38% while FLOPs fall from 12.2 to 3.66 TFLOPS and activation storage from 0.19 to 0.04 GB.","Because the method is training-free and needs only the query text and first-layer attention weights, it can be applied to existing Speech LLMs without fine-tuning.","The same pruning signal improves accuracy on DREAM-TTS and CN-College-Listen with Qwen2-Audio and on all three benchmarks with DiVA, indicating the approach transfers across models and benchmarks."],"supporting_citations":[{"why":"Supplies Qwen2-Audio, the backbone model for the main SPIRAL experiments and efficiency measurements.","marker":"[2]"},{"why":"Provides DiVA, the second Speech LLM used to test generalization, along with the audio-text alignment rationale.","marker":"[3]"},{"why":"Supplies the Whisper encoder whose frame-level outputs enable frame-wise pruning, plus the ASR metric for SPIRAL quality.","marker":"[10]"},{"why":"Represents prior vision-centric token pruning that SpeechPrune adapts from images to temporally ordered speech tokens.","marker":"[11]"},{"why":"Motivates the one-second frame size in Phase 1 via 1-2 Hz delta-band speech-perception findings.","marker":"[21]"},{"why":"Source of the DREAM-TTS conversion setup and a short-audio comparison point for long-form generalization.","marker":"[8]"},{"why":"Provides the DREAM dialogue comprehension texts behind the DREAM-TTS generalization benchmark.","marker":"[23]"},{"why":"Provides the CN-College-Listen English listening-comprehension samples used in generalization tests.","marker":"[24]"},{"why":"Synthesizes the SPIRAL spoken samples, giving the benchmark its long-form audio.","marker":"[12]"}],"fun_headline_variants":["Pruning audio tokens boosts speech retrieval accuracy 29 points","Training-free token pruning cuts compute 70%, keeps accuracy","Even 80% pruning keeps speech retrieval accuracy intact","SpeechPrune: pruning tokens lifts retrieval accuracy 29 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains rest on the assumption that the text query, including the four multiple-choice options, is worded closely enough to the spoken answer that cosine similarity to the query locates the right audio segment.","fun_headline_variants_meta":{"raw":{"variants":["Pruning audio tokens boosts speech retrieval accuracy 29 points","Training-free token pruning cuts compute 70%, keeps accuracy","Even 80% pruning keeps speech retrieval accuracy intact","SpeechPrune: pruning tokens lifts retrieval accuracy 29 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000362,"raw_usage":{"total_tokens":1951,"prompt_tokens":941,"completion_tokens":1010,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":943}},"tokens_in":557,"tokens_out":1010,"duration_ms":9543,"temperature":1.0,"reasoning_tokens":943,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:23:10.873602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SpeechPrune on SPIRAL-style questions where the correct answer is paraphrased rather than stated in the audio, or on open-ended questions with no answer options; if the 29-percentage-point gain at 20% pruning shrinks to parity with random pruning, the query-option overlap is what carries the method.","supporting_citations":[{"cited_title":"Cortical oscillations and speech processing: Emerging computational principles and operations,","cited_arxiv_id":null,"evidence_quote":"Motivates the one-second frame size in Phase 1 via 1-2 Hz delta-band speech-perception findings."},{"cited_title":"Dream: A challenge data set and models for dialogue- based reading comprehension,","cited_arxiv_id":null,"evidence_quote":"Provides the DREAM dialogue comprehension texts behind the DREAM-TTS generalization benchmark."},{"cited_title":"WavLLM: Towards robust and adaptive speech large language model,","cited_arxiv_id":null,"evidence_quote":"Provides the CN-College-Listen English listening-comprehension samples used in generalization tests."},{"cited_title":"Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models,","cited_arxiv_id":null,"evidence_quote":"Synthesizes the SPIRAL spoken samples, giving the benchmark its long-form audio."}],"review_version":1}