{"id":"47ae0d49-ce51-4ed1-9029-192215567268","arxiv_id":"2504.15509","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An offline-trained speech LLM with boundary-aware CIF speech prompts and test-time wait-k decoding achieves better quality-latency trade-offs in simultaneous speech-to-speech translation than StreamSpeech on CVSS-C.","lead":"This paper builds a speech-to-speech translation system that starts translating while the speaker is still talking, using a frozen 7B or 8B text LLM fed with streaming speech. It reports better quality at similar latency than the previous non-LLM state of the art on three language pairs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central assumption that CIF prompt length tracks the source-prefix length is untested; an oracle-prefix comparison would settle whether it is the true bottleneck.","rationale":"The paper's headline contribution is that boundary-aware CIF prompts unlock simultaneous inference of an offline-trained speech LLM, yielding a better quality-latency trade-off than StreamSpeech. The empirical comparison to StreamSpeech is informative but not perfectly controlled because SimulS2S-LLM also uses a 7B/8B pretrained text LLM, whereas StreamSpeech does not; however, the boundary-unaware ablation with the same LLM isolates the proposed mechanism and shows a large gain, so the confound does not by itself invalidate the central claim. The more load-bearing assumption is that CIF, run on partial chunks, produces a prompt whose length is an accurate count of source semantic units read so far. If that fails, the entire wait-k bookkeeping in Algorithm 1 loses its meaning, and the LLM is prompted with a prefix that is not semantically aligned to its generation position. This is exactly the mechanism the paper claims is new, and it is the place where the argument is least supported: there is no alignment-quality evaluation, no comparison against an oracle source-prefix version, and no analysis of early/late firing rates. The proposed concrete test is feasible because CVSS-C is derived from CoVoST 2, so source transcripts are available; forced alignment can produce reference source-token boundaries, and the oracle-prefix condition directly upper-bounds the achievable quality under the paper's own wait-k framework. If the oracle-prefix gap is small, the concern lands as a minor qualification; if it is large, the reported trade-off gains may not transfer beyond the tested language pairs and the method's claimed generality is weakened. Since the reader already marked the paper CONDITIONAL and identified the same weak assumption, no verdict change is needed; the additional value here is a sharper, directly executable test of that assumption.","tokens_in":19525,"tokens_out":12238,"duration_ms":125195,"concrete_test":"Use CVSS-C source transcripts to build oracle-prefix prompts: for each wait-k step, feed the same frozen LLM a prompt derived from the true source-text prefix of length Lp (obtained via forced alignments or reference word boundaries) instead of the CIF speech prompt, keeping the identical wait-k schedule and speech generator. Also compute agreement between CIF firing positions and reference source-token boundaries on partial utterances. If CIF-driven ASR-BLEU is within about 1 BLEU of the oracle-prefix result at matched ATD, the concern is resolved; if it is several BLEU lower, boundary noise is an active bottleneck and the central claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the CIF module producing, on every partial speech chunk, a speech prompt whose length Lp equals the number of source-language semantic units that have been fully read, so that the wait-k relation Lgen = Lp - Lprev - K + 1 in Algorithm 1 (Section 3.4) gives the frozen LLM a prompt that behaves like a text prefix. This assumption is only trained through a global quantity loss on complete utterances (Equation 2, |sum_t alpha_t - N|) plus the downstream S2TT cross-entropy; there is no per-boundary supervision, and the paper reports no diagnostic for how often CIF fires early or late on partial inputs. The chunk-masked streaming encoder ensures e_t and alpha_t do not depend on future frames, but the boundary decision itself is a hard threshold crossing that commits a split alpha_{t,1}/alpha_{t,2} before later evidence arrives. If alpha_t on partial chunks is miscalibrated, Lp is not a true source-prefix count, and the LLM receives a mismatched prefix; then the reported +3-4 ASR-BLEU over StreamSpeech at similar ATD could be specific to the relatively monotonic Es/Fr/De-to-English test sets rather than a general property of the proposed boundary-aware mechanism. The paper's own limitations (items 2 and 5) acknowledge this scope but do not test the boundary-quality assumption directly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SimulS2S-LLM, an offline-trained speech LLM system for simultaneous speech-to-speech translation (Simul-S2ST). The architecture combines a streaming acoustic encoder, a CIF module that extracts boundary-aware speech prompts, a frozen text LLM (BLOOMZ-7B or Llama3-8B), and a causal speech generator that predicts target-language discrete speech tokens. The system is trained in two stages: first a speech-to-text translation stage with cross-entropy and a CIF quantity loss, then a CTC-based speech-token stage with a weighted-sum of multi-layer LLM hidden states. At inference, a test-time wait-k policy controls the number of LLM tokens generated from each new speech chunk, and an incremental beam search with an n-gram speech-token LM is used for streaming speech-token decoding. Experiments on CVSS-C Es-En, Fr-En, and De-En report ASR-BLEU/ATD trade-offs against StreamSpeech and a boundary-unaware SimulS2S-LLM baseline, plus ablations on hidden-state layers and speech-token decoding, and BLASER 2.0 scores.","tokens_in":19767,"tokens_out":7289,"duration_ms":65992,"significance":"If the results hold, the paper makes a useful contribution: it is, to the best of the authors' and my knowledge, the first work to apply LLMs to Simul-S2ST while keeping the model trained offline, and it shows a clear quality-latency improvement over a boundary-unaware version of the same architecture. The strength of the paper is the internal control: the boundary-unaware SimulS2S-LLM shares the same LLM, encoder, and speech generator, so the ablation isolates the effect of CIF-based boundary-aware prompts. Evaluation uses held-out CVSS-C test sets with standard external metrics (SacreBLEU, ATD, AL, BLASER 2.0), and computation-aware latency numbers are reported in the appendix. The main risks are that the core CIF-boundary assumption is under-tested and that the headline comparison to StreamSpeech is confounded by the LLM's text pretraining.","major_comments":[{"comment":"The symbol N is used inconsistently. In Fig. 1, N is the length of the target-language token sequence [y_1,...,y_N], while the text of Section 3.2 says the quantity loss L_qua aligns accumulated weights with the source text length N. This distinction is not cosmetic: the CIF prompt length Lp in Algorithm 1 is precisely what drives the wait-k relation Lgen = Lp - Lprev - K + 1. If the quantity loss is computed against target length rather than source length, Lp is not a source-prefix count and the core mechanism is unmoored. Please define separate symbols for source and target lengths, state explicitly which length is used in Eq. (2), and report the empirical relation between Lp and the actual number of source words read on partial inputs.","section":"Section 3.2, Eq. (2), and Fig. 1"},{"comment":"The inference procedure assumes that the CIF module, run on partial chunks of streaming speech, produces speech prompts whose length Lp equals the number of source-language semantic units read so far. The training signal, Eq. (2) plus the S2TT cross-entropy, supervises only the utterance-level total count, not per-boundary firing decisions on partial inputs, and the paper reports no diagnostic for early or late CIF firing. Since a mismatched prefix would degrade the frozen LLM's predictions, please add an oracle-prefix experiment that feeds the LLM true source prefixes of the corresponding length, or report CIF fire-time statistics against actual source word boundaries. The paper's own Limitations items 2 and 5 acknowledge scope limitations but do not address this boundary-quality assumption, which is load-bearing for the central claim.","section":"Section 3.4 and Algorithm 1"},{"comment":"The headline comparison against StreamSpeech is not fully controlled. StreamSpeech is trained from scratch on CVSS-C without an LLM, whereas SimulS2S-LLM uses frozen BLOOMZ-7B or Llama3-8B backbones with large text pretraining; the abstract's phrase 'same training data' does not control for pretraining. The boundary-unaware SimulS2S-LLM ablation is the appropriate control and does support the method, but the reported 3-4 point ASR-BLEU advantage over StreamSpeech could be largely attributable to the text prior rather than to the proposed simultaneous-inference mechanism. Please either add a non-LLM architecture control trained under the same recipe, or present the StreamSpeech comparison explicitly as an external benchmark and base the method-specific claim on the boundary-aware versus boundary-unaware comparison.","section":"Section 5.1, Fig. 4, and Table 1"}],"minor_comments":[{"comment":"There is a typo: 'Henceence, the extensive comparisons' should read 'Hence, the extensive comparisons'.","section":"Section 5.1"},{"comment":"The section title 'Compution-aware Latency Results' contains a typo and should read 'Computation-aware Latency Results'.","section":"Appendix D"},{"comment":"The notation E:(n+1)∗c is ambiguous; please define E, c, n, y, Lmax, K, and Final before the pseudocode, and specify the shapes of the chunked encoder output.","section":"Algorithm 1"},{"comment":"The statement that past key-value caches including positional information are updated in parallel before LLM generation would benefit from a concrete description of how positional encodings change when Lp grows; as written, the reader cannot verify the claimed latency behavior.","section":"Section 3.4"},{"comment":"The '0.5 weight' for the speech-token n-gram LM shallow fusion should be named as a hyperparameter and listed with the other hyperparameters in Appendix B.","section":"Section 4.2"},{"comment":"The caption refers to panels (a), (b), and (c), but panel (c) is only described as an 'overall illustration'; please label the speech-prompt length and hidden-state indices in each panel so the wait-3 example is legible.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the novelty claim of being the first LLM-based Simul-S2ST system is plausible and the internal ablations are credible. The main risk is the under-specified CIF-boundary assumption and the N inconsistency in Eq. (2), both of which should be resolvable with additional analysis and a clarifying rewrite. The comparison to StreamSpeech should also be positioned as benchmark comparison rather than a controlled experiment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is the first paper to get a decoder-only speech LLM to do simultaneous S2ST, and it does so with a simple test-time wait-k policy plus CIF-derived boundary-aware prompts. That is a genuine new result for the speech translation subfield, and the paper is worth engaging.\n\nWhat is new and good: the combination of known pieces in a new application, with offline training so the LLM is not restricted to streaming; incremental beam search for speech tokens; and well-designed ablations. Boundary-aware prompts beat boundary-unaware ones by roughly 4 ASR-BLEU, multi-layer hidden states help by about a point, and n-gram fusion helps by about 1.6 points. The limitations section is honest about moderate-latency scope, compute overhead, and the lack of an LLM-based baseline. Numerical values for all plots are in tables.\n\nThe soft spots are in proportion. The main comparison with StreamSpeech is not fully controlled: SimulS2S-LLM gets a 7B/8B text LLM with a large amount of text knowledge that StreamSpeech lacks, so part of the gain is likely pretraining advantage rather than the streaming mechanism. The abstract says 3 points while Section 5.1 says approximately 4, which is minor but sloppy. There are no error bars or significance tests over the test sets.\n\nThe stress-test concern is legitimate. The wait-k relation in Algorithm 1 depends on CIF prompt length actually tracking the semantic prefix length of the source. CIF is only supervised through a global quantity loss on complete utterances, not per-boundary, and the paper reports no diagnostic for how often boundaries fire early or late on partial inputs. If CIF boundaries are noisy, the frozen LLM receives mismatched prefixes, and the reported gains could be partly specific to the three relatively monotonic European language pairs tested. An oracle-prefix comparison, where you feed the true source prefix instead of the CIF-derived one, would settle this cleanly and is the missing experiment. It is not fatal: the boundary-unaware baseline being much worse is real evidence that boundary information matters, but it does not prove the prompt lengths are calibrated.\n\nWho is this for: people working on streaming speech LLMs or simultaneous S2ST, and anyone building practical live interpretation systems. It deserves a serious referee. The central result is likely to hold qualitatively even if the exact gain over StreamSpeech is inflated. I would recommend engaging with it and asking for the oracle-prefix diagnostic, confidence intervals, and a clarification of which number in the abstract is intended.","headline":"First LLM-based simultaneous speech-to-speech translation system with a clean recipe and honest limitations; the headline gain over StreamSpeech is real but partly confounded by LLM text pretraining, and the CIF boundary assumption deserves a direct diagnostic.","tokens_in":20372,"tokens_out":1500,"would_cite":true,"duration_ms":14090,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An offline-trained speech LLM can be made to stream speech-to-speech translation by feeding it boundary-aware speech prompts from CIF and decoding with a test-time wait-k policy; on CVSS-C it beats existing same-data simultaneous systems…","keywords":["simultaneous speech-to-speech translation","speech large language models","continuous integrate-and-fire","wait-k policy","discrete speech tokens","streaming inference","CVSS corpus","boundary-aware speech prompts"],"falsifier":"Take the trained system and perturb the CIF firing times on the first chunks, shifting or dropping a fraction of prompt boundaries, and measure the ASR-BLEU change; if a small perturbation erases the advantage over StreamSpeech, then CIF boundary accuracy is the causal ingredient. A complementary check is to compute alignment error between CIF firing times on streaming prefixes and oracle word boundaries on CVSS-C test audio.","tokens_in":19263,"feed_emoji":"🗣️","tokens_out":13763,"duration_ms":108119,"temperature":0.7,"pith_summary":"SimulS2S-LLM sets out to show that a speech-conditioned large language model trained offline on complete utterances can still translate speech to speech while the input is arriving. The enabling idea is to feed the frozen LLM a boundary-aware speech prompt built by continuous integrate-and-fire, so the number of prompt vectors available from a chunk of audio tracks how many source-language tokens have been spoken. At test time a wait-k policy keeps LLM generation k tokens behind the growing prompt, and discrete speech tokens are decoded from a weighted sum of the LLM's hidden states and turned into audio by a vocoder. On CVSS-C Spanish-English, French-English, and German-English the paper reports a better quality-latency trade-off than existing simultaneous speech-to-speech methods trained on the same data, for example about 3 ASR-BLEU points higher at similar latency. If the claim is right, a single offline-trained speech LLM can serve both offline and streaming tasks without retraining.","feed_headline":"Boundary-aware prompts let an offline speech LLM translate live","feed_subtitle":"Cutting the streaming speech prompt at word boundaries lets a frozen LLM run wait-k decoding and keep offline quality.","key_machinery":"The carrying object is the boundary-aware speech prompt extracted by CIF (continuous integrate-and-fire). For each encoder frame $e_t$ a scalar alpha_t is accumulated from left to right; when the running sum passes 1.0 the current weight is split so that exactly 1.0 fires one prompt vector $p_i$, the weighted frame sum is emitted, and accumulation resets. Because the CIF is trained with a quantity loss matching total accumulated weight to the source text length $N$, the number of prompt vectors read from a partial audio chunk is a proxy for how many source tokens have been spoken, which is what lets the offline-trained, frozen LLM treat the partial prompt like a text prefix. This makes the test-time rule $L_{gen} = L_p - L_{prev} - K + 1$ (Algorithm 1) work: the LLM stays $K$ tokens behind the prompt, exactly as in text wait-k simultaneous translation.","core_discovery":"The paper's central claim is that the obstacle to streaming a speech LLM is not the decoder-only architecture itself but the mismatch between the full-speech prompt seen in offline training and the partial prompt available during simultaneous inference. SimulS2S-LLM removes that mismatch with CIF: encoder frames are weighted and integrated left-to-right, and each time the accumulated weight reaches 1.0 the weighted frame sum fires as one prompt vector, so the prompt length sits at the source-text-token level. Inference is then driven by the rule $L_{gen} = L_p - L_{prev} - K + 1$ from Algorithm 1, which keeps the LLM exactly $K$ tokens behind the growing speech prompt and reproduces the behaviour of text-based wait-k. Output speech is generated from a weighted sum of multi-layer LLM hidden states by a causal speech generator using CTC, an incremental beam search, and a speech-token n-gram language model, with a pre-trained vocoder synthesising audio. The paper reports that on CVSS-C Es-En, Fr-En, and De-En this gives a better ASR-BLEU-versus-latency trade-off than StreamSpeech and than a boundary-unaware downsampling baseline, with the boundary-aware prompts contributing roughly 4 BLEU points at matched latency.","pith_inferences":["The paper does not measure how accurate CIF boundaries are on partial chunks; a natural extension would be to compare CIF firing times against oracle source-word boundaries on streaming prefixes, since boundary error on early chunks should predict how much of the reported quality-latency gain transfers to noisier audio.","The recipe is not translation-specific, so the same boundary-aware prompt plus wait-k could unlock streaming for other speech-conditioned LLM tasks such as live speech continuation or speech question answering; the paper only evaluates translation and does not claim these transfers.","The fixed wait-k already needs larger $K$ for German-English than for Spanish/French-English, which suggests that for language pairs with heavier reordering an adaptive or learned latency policy would extend the method beyond what the paper tests.","The computation-aware latency in Appendix D shows the frozen LLM itself is a large part of the measured delay; as fast serving or speculative decoding improves, the same algorithm's quality-latency curve would shift left, so part of the reported trade-off is a hardware statement."],"forward_implications":["Changing $K$ at test time moves the system along the quality-latency curve without retraining, so one offline-trained checkpoint can serve many latency budgets.","Because the CIF prompt behaves like a source-text prefix, the wait-k tail beam search, incremental beam search, and speech-token n-gram shallow fusion all apply cleanly to speech output.","The frozen LLM is never trained for streaming, so the same model retains whatever non-streaming capabilities the base text LLM has; streaming behaviour is an inference-time property.","On the three language pairs tested, SimulS2S-LLM improves ASR-BLEU by roughly 3-4 points over StreamSpeech at comparable latency, and the advantage survives computation-aware latency measurement in the appendix.","Scaling should be additive: the paper states that using a larger text LLM or more speech-to-speech training data should further improve the generated speech quality and the trade-off."],"supporting_citations":[{"why":"Supplies the test-time wait-k read-write policy and the tail beam search that SimulS2S-LLM adapts to speech output.","marker":"Ma et al., 2018"},{"why":"Supplies the CIF mechanism that extracts token-level boundary-aware speech prompts from streaming encoder frames.","marker":"Dong and Xu, 2020"},{"why":"Provides the CIF scalar-weight variant and the argument that extremely low latency is unsuitable for offline-trained simultaneous models.","marker":"Deng and Woodland, 2024b"},{"why":"Supplies the fixed downsampling method used to build the boundary-unaware speech prompt baseline.","marker":"Fathullah et al., 2024"},{"why":"Inspires the up-sampling of hidden states with a CTC objective for predicting target speech tokens from LLM states.","marker":"Fang et al., 2024"},{"why":"Provides StreamSpeech, the strong non-LLM simultaneous speech-to-speech baseline that SimulS2S-LLM is compared against and outperforms.","marker":"Zhang et al., 2024"},{"why":"Provides the CVSS-C speech-to-speech translation corpus used for all reported experiments.","marker":"Jia et al., 2022b"},{"why":"Supplies the mHuBERT discrete semantic speech tokens used as prediction targets for the target-language speech.","marker":"Popuri et al., 2022"},{"why":"Supplies the pre-trained HiFi-GAN vocoder that synthesises audible speech from the predicted discrete tokens.","marker":"Kong et al., 2020"},{"why":"Supplies BLOOMZ-7B1, the frozen text LLM used as the generation backbone for Spanish-English and French-English.","marker":"Scao et al., 2022"}],"fun_headline_variants":["Boundary-aware prompts let speech LLMs translate live","Speech LLM streams via boundary-aware prompt cutting","Wait-k speech translation without retraining: prompt boundaries","Offline speech LLM goes live with boundary-aware prompts","Streaming speech translation: cut prompts at word boundaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the assumption that CIF, when it only sees part of the audio, still counts out prompt vectors whose cumulative length tracks the semantic prefix of the source, so the frozen LLM is always handed the right prefix; if boundary counts drift on partial input, the quality-latency gains would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Boundary-aware prompts let speech LLMs translate live","Speech LLM streams via boundary-aware prompt cutting","Wait-k speech translation without retraining: prompt boundaries","Offline speech LLM goes live with boundary-aware prompts","Streaming speech translation: cut prompts at word boundaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1617,"prompt_tokens":1041,"completion_tokens":576,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":500}},"tokens_in":657,"tokens_out":576,"duration_ms":5691,"temperature":1.0,"reasoning_tokens":500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:24:37.235214+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained system and perturb the CIF firing times on the first chunks, shifting or dropping a fraction of prompt boundaries, and measure the ASR-BLEU change; if a small perturbation erases the advantage over StreamSpeech, then CIF boundary accuracy is the causal ingredient. A complementary check is to compute alignment error between CIF firing times on streaming prefixes and oracle word boundaries on CVSS-C test audio.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained HiFi-GAN vocoder that synthesises audible speech from the predicted discrete tokens."}],"review_version":1}