{"id":"b2667981-0e5a-4d92-adfa-763254bc63aa","arxiv_id":"2501.02370","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Across controlled ASR and speech translation experiments, dense feature prepending does not outperform cross-attention in quality and is slightly slower and more memory hungry.","lead":"Researchers trained speech-to-text models from scratch in three architectures, prepending speech features to a decoder versus using cross-attention, and found no consistent quality advantage for prepending. The result matters because prepending is the dominant way to connect speech to large language models, and the paper suggests the more traditional cross-attention design is at least as good and more efficient at the scale tested.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central negative claim treats p>0.05 as proof of parity; without equivalence margins or confidence intervals, 'no clear advantage' is not established by the reported tests.","rationale":"The paper is a carefully controlled comparison with public benchmarks and released code; the authors also run significance tests, which is more than many negative-result papers do. The reader's concern about shared hyperparameters and single runs is valid, but I see a more fundamental issue that makes the central claim under-supported independent of tuning. The abstract's conclusion is an equivalence-style negative claim: 'our results do not indicate a clear advantage of DFP over cross-attention.' The supporting evidence rests largely on non-significant p-values in Table 1, especially for the Conformer-CTC rows where differences are declared not significant and then interpreted as parity. Statistically, failing to reject the null is not evidence for the null; with small test sets and no confidence intervals, the true difference could be large in either direction. The paper does not define what magnitude of difference would constitute a 'clear advantage,' nor does it provide interval estimates for the key comparisons. This is a load-bearing gap: even if every experimental detail were perfect, the conclusion as stated would not follow from the reported statistics. I therefore agree with the CONDITIONAL verdict: the paper deserves publication as an empirical study, but the central claim should be reworded to 'we find no statistically significant advantage' or, better, supported by an equivalence analysis with pre-specified margins. The scale limitation is explicitly acknowledged in §7 and is not a flaw in itself; the missing equivalence/confidence analysis is the main obstacle to accepting the abstract at face value.","tokens_in":16740,"tokens_out":6655,"duration_ms":63161,"concrete_test":"Using the released code and checkpoints, compute 95% bootstrap confidence intervals for the per-language difference in WER/BLEU between cross-attention and decoder-prepend for the Conformer-CTC configuration in Table 1 (lines 4 vs 5, and 4.1 vs 5.1 for compressed variants), resampling test-set segments as in the paper's significance tests. If any interval has width exceeding a pre-specified margin (e.g., ±0.5 BLEU or ±1 WER), the data are compatible with a meaningful advantage for either architecture, and the abstract's 'no clear advantage' claim must be weakened or qualified. If all intervals lie within the margin, the negative claim gains proper support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §5.1, after Table 1 (lines 4–5), the paper states that for Conformer with auxiliary CTC, 'None of these differences are statistically significant with p < 0.05' and then concludes that 'decoder-prepend is on par with cross-attention.' This is the core inferential step supporting the central claim of 'no clear advantage of DFP over cross-attention.' The reasoning is invalid as stated: a non-significant difference does not demonstrate that the true difference is small or zero. It only indicates that the test lacked evidence to reject the null. With typical test-set sizes and single training runs, the data may be consistent with a wide range of true differences. The paper provides no pre-specified equivalence margin (e.g., a WER or BLEU difference considered negligible) and no confidence intervals for the cross-attention vs. decoder-prepend differences. For a negative/equivalence claim, the absence of a statistically significant difference is insufficient; what is needed is evidence that the difference is bounded within a meaningful interval. Without that, the central claim could be an artifact of low statistical power. This concern does not depend on hyperparameter tuning or scale; it affects the interpretation of the reported numbers even if the experimental design is otherwise flawless.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a controlled empirical comparison of three ways to integrate speech into sequence-to-sequence text models: a standard encoder-decoder with cross-attention, a decoder-only model in which downsampled audio is prepended to text tokens, and a decoder-prepend model in which a speech encoder's representations are prepended. All models are trained from scratch on MuST-C v1.0 and CoVoST2, with Transformer and Conformer encoders, and are evaluated on ASR and ST across monolingual, bilingual, and multilingual settings, including CTC auxiliary loss and compression, sequence-level knowledge distillation, generation speed, GPU memory, and a causal-masking ablation. The main empirical conclusion is that, at the tested scale, decoder-prepend does not show a clear advantage over cross-attention, both outperform decoder-only models on quality and efficiency, and CTC compression is more favorable to decoder-prepend than to cross-attention.","tokens_in":16965,"tokens_out":13907,"duration_ms":112052,"significance":"The study is a timely and useful counterpoint to the common assumption that dense feature prepending is the preferred way to connect speech encoders to LLMs. Its strengths include a deliberately controlled setup (same data, same training recipe, models trained from scratch), evaluation over 12 language directions and two tasks, multiple configurations (Transformer/Conformer, CTC compression, seqKD), significance tests for the main comparisons, and released code under a permissive license. If the central negative claim is accepted, the paper provides practical guidance that cross-attention is a safe, more efficient default at this scale. The main fragility is inferential: the evidence is stronger for 'we did not find an advantage' than for 'the architectures are on par,' and the paper's own wording alternates between the two; this needs to be resolved in revision.","major_comments":[{"comment":"The sentence 'None of these differences are statistically significant' is immediately followed by 'The above results show that decoder-prepend is on par with cross-attention.' This is not a valid equivalence inference: a null result from a significance test only means the test did not detect a difference, and with single runs it says nothing about the size of the true difference. The data could be consistent with a meaningfully worse or better decoder-prepend under a different sample or seed. To support the parity language in the Introduction ('overall similar results') and §5.1, the authors should add pre-specified equivalence margins or confidence intervals for the WER/BLEU differences; otherwise the conclusions should be rephrased as absence-of-evidence claims, consistent with the more cautious Abstract wording.","section":"§5.1, Table 1 (lines 4–5)"},{"comment":"All three architectures are trained with one shared recipe (Adam, peak learning rate 2e-3, 25k warmup, batch sizes 320k/256k frames, early stopping patience 10) and there is no per-architecture hyperparameter tuning. A shared recipe is a reasonable control, but for a comparative negative claim the absence of any sensitivity analysis leaves open the possibility that the architecture ranking would change under different hyperparameters. Please report a small robustness check (e.g., learning rate or warmup variation) for the central comparison, or explicitly discuss the risk and the evidence that the chosen budget is fair to all three architectures.","section":"§4.3"},{"comment":"Most reported results are single training runs, and the significance tests (bootstrap resampling for ASR, approximate randomization for ST) quantify only test-set sampling variability, not training stochasticity. The p-values and †/‡ markers in Table 1 are therefore insufficient to support statements such as 'None of these differences are statistically significant' as evidence about architecture superiority. The authors should either provide multiple seeds for the key cross-attention vs. decoder-prepend comparisons (at least Table 1 lines 4–5) or explicitly state that the significance tests ignore training variance.","section":"§4.3, Table 1"},{"comment":"The claim that 'decoder-prepend better leverages CTC compression' is based on the observation that compression degrades cross-attention (WER 19.6→21.8, BLEU 29.7→28.5) while decoder-prepend is unchanged (19.9→19.9, 29.7→29.7). This is a difference-in-differences claim, but no interaction test or confidence intervals are provided for those differences, and both comparisons rest on single runs. Please test the architecture-by-compression interaction directly or provide intervals, so readers can judge whether the asymmetry is beyond noise.","section":"§5.2, Table 1 (lines 4.1 and 5.1)"}],"minor_comments":[{"comment":"In the plain-text rendering, the #Params entry for 'decoder-only 18L' is not visible; please ensure every row reports parameter counts and that the table is not missing a cell.","section":"Table 1"},{"comment":"'Experiments are run on 4 Nvidia A100-40GB GPUs for about 2 days' is ambiguous about whether this is per model or for the whole suite; please state the per-model training time.","section":"§4.3"},{"comment":"The abbreviations 'compr' and 'CF-compr' are used inhomogeneously; define both in one place and use them consistently.","section":"Figure 2 and §5.2"},{"comment":"The claim that seqKD makes the target 'more monotonically aligned' to the source should cite the specific analysis in Zhou et al. (2019), not only the general seqKD method.","section":"§5.3"},{"comment":"The 8.18 BLEU drop for decoder-prepend TF on multilingual ST when causal masking is removed is striking and is attributed to longer inputs; a per-language breakdown or an analysis by input length would make the hypothesis testable.","section":"§5.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid, carefully executed empirical study and the main inferential issue is fixable. I would encourage the editor to request a revision that tightens the parity claims and adds either equivalence analysis or explicit absence-of-evidence language, plus at least a minimal robustness check. The paper's scope (training from scratch at 65–153M parameters) is clearly stated, so I would not reject on generalizability grounds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the cleanest controlled comparison I have seen of cross-attention versus dense feature prepending for speech-to-text: from-scratch training, public benchmarks, multiple tasks and language directions, plus a genuinely new ablation on causal masking for decoder-prepend. Second, the paper sometimes says the two architectures are \"on par\" when the reported tests only show differences are not significant at p<0.05. That distinction matters, and it is the main soft spot.\n\nWhat is actually new: prior comparisons used large pretrained models with confounded settings; this paper removes that dependency and gives a fair architecture-level comparison across ASR and ST, mono/bi/multilingual, with CTC compression and seqKD. The causal masking result is the most interesting finding: decoder-only models need non-causal audio masking, while decoder-prepend slightly benefits from causal masking, likely because the speech encoder already provides full attention. That is a concrete, non-obvious design guideline. The paper also reports speed and memory, and shows cross-attention is more efficient.\n\nWhat the paper does well: the experimental design is careful, the datasets and evaluation are standard, significance tests are reported, code is released, and the limitations section honestly acknowledges the single-scale, no-large-pretrained-models scope. The finding that decoder-prepend gets more benefit from CTC compression than cross-attention is a useful nuance that could guide future work.\n\nWhere it is soft: the stress-test note is on target. In 5.1, the paper writes \"None of these differences are statistically significant\" and then concludes \"decoder-prepend is on par with cross-attention.\" A non-significant difference is not evidence of parity; it could be low power. To support \"on par\" they would need pre-specified equivalence margins or confidence intervals for the differences. The abstract's phrasing \"our results do not indicate a clear advantage\" is actually fine, but the body's \"on par\" overstates the finding. This is fixable in revision, not a fatal flaw. The shared hyperparameter budget and mostly single-run results are real limitations, acknowledged by the authors, but they make the general claims weaker than the tone sometimes suggests. The small scale (65M-153M parameters) also means the conclusions should not be projected to LLM-scale DFP without caution.\n\nWho this is for: researchers working on speech-LLM integration, especially those choosing an architecture for ASR or ST. They will get concrete, reproducible evidence and a useful caution against assuming DFP is superior. It deserves a serious referee. My recommendation: send it to review, but ask the authors to tighten the parity claim, report confidence intervals or equivalence tests where feasible, and be explicit that the negative result is \"no evidence of advantage at this scale,\" not \"equivalence.\"","headline":"A clean, from-scratch comparison showing DFP does not beat cross-attention for speech-to-text at this scale, but the paper overstates 'parity' where the statistics only support 'no significant difference found.'","tokens_in":17539,"tokens_out":2596,"would_cite":true,"duration_ms":25874,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dense feature prepending offers no clear quality edge over cross-attention for speech-to-text models.","keywords":["dense feature prepending","cross-attention","decoder-only speech models","speech-to-text translation","automatic speech recognition","CTC compression","sequence-level knowledge distillation","causal masking"],"falsifier":"Train decoder-prepend and cross-attention models at a much larger scale (for instance, 1B+ parameters) on the same data, with per-architecture hyperparameter search, and compare WER/BLEU and throughput. If dense feature prepending then clearly beats cross-attention in quality or efficiency, the paper's conclusion would be limited to small from-scratch models.","tokens_in":16515,"feed_emoji":"🎙️","tokens_out":6609,"duration_ms":55660,"temperature":0.7,"pith_summary":"The paper asks whether the dominant way of feeding speech into large language models—dense feature prepending (DFP), where projected audio features are concatenated ahead of the text tokens—actually beats the classic cross-attention encoder-decoder design. To answer this, the authors train all three architectures from scratch on the same multi-language speech recognition (ASR) and speech translation (ST) datasets, with comparable parameter counts and identical training recipes. They find that DFP and cross-attention deliver statistically similar quality, while cross-attention is slightly faster and more memory-efficient; decoder-only models that skip a speech encoder are clearly worse. The paper also shows that CTC-based audio compression helps DFP more than cross-attention, and that causal masking of speech features helps decoder-prepend but hurts decoder-only models. These results matter because DFP has become the default in speech-LLM integration, and the paper indicates it is not a clear win over a simpler baseline.","feed_headline":"Prepending speech to LLMs matches, not beats, cross-attention","feed_subtitle":"Speech recognition and translation quality are on par, while cross-attention is faster and uses less memory.","key_machinery":"The comparison rests on three architectural families with identical encoder/decoder layer counts and dimensions: cross-attention (a standard encoder-decoder with cross-attention in the decoder), decoder-prepend (a speech encoder's output is concatenated with text embeddings in a decoder-only stack), and decoder-only (no speech encoder; only a length adapter before prepending). Around these, the paper varies two mechanisms: CTC compression, which merges consecutive frames with identical predictions to shorten the audio sequence, and sequence-level knowledge distillation, which replaces target translations with more monotonic synthetic translations. A final manipulated variable is the causal mask over the concatenated speech+text sequence, which either restricts speech frames to look only backward or lets them attend to all speech frames.","core_discovery":"The central claim is that, under controlled from-scratch training at the 65M–153M parameter scale, dense feature prepending (DFP) does not give a clear quality advantage over cross-attention for speech-to-text. In the Transformer setting, cross-attention is slightly better on average; in the Conformer-with-CTC setting, decoder-prepend and cross-attention are statistically indistinguishable on both ASR and ST. On efficiency, cross-attention is consistently faster and uses less GPU memory than decoder-prepend, while decoder-only models are slower, more memory-hungry, and lower in quality. Two further results qualify the picture: CTC compression is more beneficial for DFP than for cross-attention, and causal masking of the speech portion helps decoder-prepend but substantially hurts decoder-only models.","pith_inferences":["If the pattern holds at larger scale, many production speech-language systems built on DFP could switch to cross-attention without quality loss and gain throughput and memory headroom.","The causality result suggests that in DFP the speech encoder is doing the full-attention work that the decoder's causal self-attention cannot; without an encoder, full attention over speech is necessary.","The fixed training recipe leaves open that architecture-specific tuning could narrow or widen the gaps, so the conclusion is best read as 'DFP has no clear advantage under a common default recipe' rather than as a ceiling for DFP.","A direct extension would be to test instruction-formatted prompts between speech and text, since the paper notes this LLM-specific modeling choice is absent from its setting and could alter the comparison."],"forward_implications":["Cross-attention is a strong, simple baseline for any future speech-LLM integration; DFP's adoption is not justified by quality at this scale.","CTC compression is a cheap and effective technique for DFP models, giving most of the speed and memory benefit without quality loss.","Causal masking should be kept on decoder-prepend models and removed for decoder-only models; a speech encoder changes how the decoder should mask audio.","Decoder-only speech models at this scale are dominated by both cross-attention and decoder-prepend, which supports keeping a speech encoder when pursuing the LLM integration path."],"supporting_citations":[{"why":"Proposed the decoder-only architecture for speech-to-text and claimed it can match encoder-decoder models; the paper's decoder-only baseline is built on this line.","marker":"Wu et al., 2023"},{"why":"Reported decoder-only ASR models surpassing encoder-decoder ones; the paper tests this claim and finds decoder-only worse at its scale.","marker":"Gupta et al., 2024"},{"why":"Supplied the CTC-based compression method that merges frames with identical predictions, used as a key configuration.","marker":"Gaido et al., 2021"},{"why":"Introduced sequence-level knowledge distillation, which the paper applies to ST models.","marker":"Kim and Rush, 2016"},{"why":"Provided the MuST-C dataset used for bilingual and multilingual ST and English ASR.","marker":"Di Gangi et al., 2019a"},{"why":"Provided the CoVoST2 dataset with 21 non-English source languages for multilingual ASR and ST.","marker":"Wang et al., 2021"},{"why":"Proposed the Conformer encoder that the paper uses as the stronger speech encoder in the main comparison.","marker":"Gulati et al., 2020"},{"why":"Questioned DFP on the analogy with encoder-decoder models, motivating the direct comparison.","marker":"Chen et al., 2024c"}],"fun_headline_variants":["Prepending speech to LLMs? No edge over cross-attention","Cross-attention beats prepending on speed, matches on quality","Speech-to-text: Prepending loses to cross-attention in efficiency","LLM speech integration: Cross-attention just as good, faster","Why prepend speech? Cross-attention does it without the cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the same training recipe—optimizer, learning-rate schedule, batch size, and early stopping—is a fair basis for comparing the three architectures; if each architecture needs different hyperparameters to perform well, the measured gaps could be artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Prepending speech to LLMs? No edge over cross-attention","Cross-attention beats prepending on speed, matches on quality","Speech-to-text: Prepending loses to cross-attention in efficiency","LLM speech integration: Cross-attention just as good, faster","Why prepend speech? Cross-attention does it without the cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3637,"prompt_tokens":904,"completion_tokens":2733,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":2643}},"tokens_in":520,"tokens_out":2733,"duration_ms":21198,"temperature":1.0,"reasoning_tokens":2643,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:13:08.034833+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train decoder-prepend and cross-attention models at a much larger scale (for instance, 1B+ parameters) on the same data, with per-architecture hyperparameter search, and compare WER/BLEU and throughput. If dense feature prepending then clearly beats cross-attention in quality or efficiency, the paper's conclusion would be limited to small from-scratch models.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reported decoder-only ASR models surpassing encoder-decoder ones; the paper tests this claim and finds decoder-only worse at its scale."}],"review_version":1}