{"id":"eb7a2918-a4ce-4eb9-a110-d2ddbe02dc62","arxiv_id":"2505.16983","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Streaming LLMs need only fix the input-attention mismatch; group position encoding (source and target positions numbered separately) removes the need for re-encoding and outperforms specialized streaming baselines.","lead":"The paper examines why large language models lose accuracy when switched from batch to streaming use, and shows that only one of three structural conflicts matters: source tokens seeing earlier outputs. It proposes a simple position-numbering scheme, group position encoding, that avoids expensive re-encoding and runs streaming translation and speech recognition up to 11 times faster.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central diagnosis is confounded: Table 1 changes both the attention mask and the input token order between interleaved- and batch-streaming, so the +1.2-2.2 BLEU gain cannot be attributed to input-attention mismatch alone.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the Section 3, Table 1 ablation confounds input-attention masking with input token ordering. I agree with that assessment. In good faith, the paper does provide substantial support for the practical method: the re-encoding ablations show that position re-encoding and KV re-encoding add little beyond the no-re-encoding condition, the group position encoding is simple and model-agnostic, and the generalization to batch processing is tested. Those results are independent of the disputed attribution and support the practical claim that re-encoding can be dropped. However, the abstract's analytical claim that only the input-attention mismatch significantly impacts performance is load-bearing for the paper's narrative, and the reported experiment does not isolate that mismatch because both the mask and the token order change. This is not a question of internal inconsistency or bad faith; it is an uncontrolled variable in a key ablation. The proposed control experiment would settle the ambiguity. If the control supports the mask as the driver, the analytical claim stands; if it supports order, the practical method might still be useful but the diagnosis would need revision. Either way, the current evidence justifies a conditional rather than an unconditional acceptance, which matches the reader's verdict, so I recommend no change.","tokens_in":22496,"tokens_out":5716,"duration_ms":41627,"concrete_test":"Run the missing 2x2 cell: interleaved-streaming with the same interleaved source/target token order and the same interleaved position IDs as Figure 5(b), but with the causal mask modified to block source tokens from attending to target tokens (target-to-source and target-to-target attention unchanged). Train and evaluate with the same LoRA settings and wait-k values used in Table 1 (at least wait-1 and wait-3) on IWSLT-17 En-Fr and En-De. If the BLEU of this 'interleaved-blocked' condition approaches the 'Batch-streaming (No re.)' numbers (e.g., roughly 33.1 vs 30.9 for Gemma2 wait-1 En-Fr), input-attention mismatch is the driver; if it stays near 'Interleaved-streaming' (e.g., roughly 30.9), then the reported gain is due to reordering the input, and the paper's central diagnosis fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assertion is that 'only the input-attention mismatch significantly impacts performance' (Abstract), which motivates dropping re-encoding and proposing group position encoding. In Section 3, Table 1, the comparison supporting this is 'Interleaved-streaming' vs. 'Batch-streaming (No re.)'. The paper states that batch-streaming (No re.) uses the same positional encoding as interleaved-streaming but 'maintains the batch-processing input format' (Appendix B.3, Figure 5c), i.e., all source tokens precede all target tokens, whereas interleaved-streaming alternates source and target chunks (Figure 5b). Thus two variables change simultaneously: (i) the mask blocks source-to-target attention, and (ii) the token order changes, which under the causal mask alters what earlier tokens attend to. The observed +1.2-2.2 BLEU gain is attributed entirely to (i), but no condition keeps interleaved order while only blocking source-to-target attention. If the gain actually comes from (ii), the claimed diagnosis is wrong: position/order effects would be substantial, and the conclusion that re-encoding outputs is unnecessary would rest on a different, unstated mechanism. The practical group-streaming method may still work, since it adopts source-first order plus the blocked mask, but the paper's analytical central claim is not established by the reported experiment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how to adapt batch-pretrained decoder-only LLMs to streaming tasks. It identifies three mismatches between streaming and batch processing: input-attention, output-attention, and position-ID mismatches. Through stepwise ablations on IWSLT-17 En-Fr and En-De translation with Gemma2-2B and Phi3-mini, it reports that blocking source-to-target attention yields the largest gains (up to +2.20 BLEU), while position re-encoding and KV-cache re-encoding give only small improvements (at most about 0.28 BLEU). From this it concludes that re-encoding outputs is largely unnecessary. To explain the result, the paper analyzes position encoding in streaming settings, argues that relative order within source and target groups matters more than absolute position, and proposes group position encoding: source tokens keep IDs 0,1,2,... and target tokens start at a hyperparameter phi, with source-first input order and no source-to-target attention. The method is evaluated on text translation and streaming ASR (LibriSpeech), compared against DST, SimulMask, CAAT, and wav2vec-S baselines, and reported to achieve up to 11.3x throughput gains over re-encoding baselines.","tokens_in":22798,"tokens_out":8523,"duration_ms":69529,"significance":"If the central diagnostic claim is established, this would be a practically valuable result: it would show that streaming capability can be added to off-the-shelf instruct LLMs via LoRA fine-tuning with a changed attention mask and position numbering, avoiding expensive re-encoding while preserving quality. The paper has clear strengths: Table 1 is a clean stepwise decomposition of re-encoding effects (with the caveat below), the insensitivity of the hyperparameter phi is tested over a wide range (0 to 512 for translation, 0 to 2048 for ASR) across three model families, the experiments span two modalities, the code is promised publicly, and the efficiency numbers in Appendix F are concrete and machine-checkable. The claim that re-encoding is unnecessary for quality rests on comparisons that are mostly well controlled within the batch-streaming framework; the weak link is the attribution of the interleaved-versus-batch gain to the input-attention mismatch, which is confounded by token ordering.","major_comments":[{"comment":"The comparison between 'Interleaved-streaming' and 'Batch-streaming (No re.)' varies two factors simultaneously. In interleaved-streaming the input alternates source and target chunks, whereas batch-streaming (No re.) places all source tokens before all target tokens; under a causal mask this reordering itself changes which source tokens can attend to which previous tokens, independently of the explicit blocking of source-to-target attention. The reported +1.2 to +2.2 BLEU gains are therefore not uniquely attributable to removing the input-attention mismatch. Please add a control that keeps the interleaved token order and blocks source-to-target attention with a custom mask (or, conversely, keeps source-first order while allowing source-to-target attention). Without such a control, the abstract's claim that 'only the input-attention mismatch significantly impacts performance' is not established, and the paper's diagnosis of the three mismatches rests on an unproven attribution.","section":"Section 3, Table 1; Appendix B.3, Figure 5(b)-(c)"},{"comment":"The comparisons with SimulMask-ALiBi, DST, CAAT, and wav2vec-S are presented as evidence that group-streaming LLMs 'consistently outperform' existing approaches, but the figure appears to plot points from prior publications with no indication of matched training data, model sizes, decoding settings, or hyperparameters, and no error bars or significance tests are provided. This severely limits the strength of the empirical superiority claim. Please provide a table with the exact baseline numbers, their sources, and a description of the evaluation protocol, or temper the claim accordingly.","section":"Section 5, Figure 4"}],"minor_comments":[{"comment":"The position-removal simulation by assigning a constant position ID of 0 to all tokens is a reasonable proxy, but it should be described as a proxy rather than as directly removing the positional encoding module; a direct ablation (e.g., disabling RoPE or randomizing position IDs) would further support the relative-order conclusion.","section":"Section 4.1, Table 2"},{"comment":"The notation for time indices is sometimes loose: for example, Eq. (2) uses Y_{<t'} and Eq. (4) uses X_{\\le t} without explicitly defining how t and t' relate to the wait-k policy; please make the connection explicit.","section":"Section 3, Equations (1)-(4)"},{"comment":"There is a formatting/typographical error in the first row of Table 2: '540.7640.68' should presumably read '5  40.76  40.68...'; please correct it.","section":"Appendix D.3, Table 2"},{"comment":"The sentence 'interleaved position encoding can be viewed as a special case of group position encoding' is confusing because interleaved mode does not use grouped position IDs; please rephrase to clarify the intended relationship.","section":"Section 4.2"},{"comment":"The efficiency claim 'up to 11.3x' is based on a single filtered subset with one model; please state in the main text that this is a case study and clarify the exact experimental conditions there.","section":"Appendix F, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The practical method is promising and the results are likely reproducible, but the central analytical claim needs one additional control experiment. The confound in Table 1 is straightforward to fix and does not, in my view, invalidate the proposed method; it does, however, undermine the paper's stated diagnosis as written. I would be willing to accept after the authors add the missing control and tighten the baseline comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper makes two things worth your time: a clean efficiency result and a simple model-agnostic recipe. It shows that for wait-k streaming translation and ASR, you can drop all the KV/position re-encoding that prior streaming-LLM systems treat as necessary, fine-tune an instruct LLM with a small LoRA rank, use a source-first input format with a blocked source-to-target mask, and independently assign position IDs to source (0..) and target (phi..). Throughput goes up 5-11x with no quality loss. That is a genuinely useful result, and the group position encoding idea is easy to adopt. The empirical work is mostly sound. The within-batch-streaming ablations are clean: adding position re-encoding adds at most ~0.1-0.2 BLEU, and KV re-encoding adds ~0.1-0.2 more. That supports the practical claim that re-encoding is unnecessary, regardless of what mechanism explains the gap. The robustness of phi is nicely demonstrated over 0 to 512, and the edge experiment with very large phi (up to 50k) shows the expected breakdown, which is consistent with the paper's account. The soft spot is the attribution of the gain to the input-attention mismatch. Table 1 compares interleaved-streaming (interleaved order, causal mask) to batch-streaming-no-reencoding (source-first order, blocked mask). Both the token order and the attention pattern change, so the +1.2-2.2 BLEU cannot be pinned on the attention mismatch alone. A control that keeps interleaved order while blocking source-to-target attention would settle it. Without that, the paper's 'first comprehensive analysis' claim is a bit overstated, and the central diagnosis is not fully established. The method itself probably still works since it adopts the same source-first order plus blocked mask, but the explanation of why needs that control. Also worth noting: the position-removal experiment replaces positions with constant ID 0, which is a proxy, not a true removal; and Figure 4's baseline comparisons could use more details on training/eval. Minor, but should be in the revision. Who should read this: anyone doing streaming translation, ASR, or decoder-only streaming in general. The paper is a solid candidate for serious review. It deserves referee time; just ask the authors to fix the confound and the proxy in revision. My verdict: send it to review, with a request for the isolating control.","headline":"Genuinely useful streaming-LLM paper with a simple group position encoding recipe that drops expensive re-encoding; the central mismatch analysis has a real but fixable confound.","tokens_in":681,"tokens_out":787,"would_cite":true,"duration_ms":39112,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that blocking input-to-output attention and giving source and target tokens independent position IDs—group position encoding—is enough to make an existing LLM stream without re-encoding.","keywords":["streaming LLM","group position encoding","input-attention mismatch","wait-k policy","simultaneous translation","automatic speech recognition","position encoding","re-encoding"],"falsifier":"Run the interleaved-streaming setting but modify only the attention mask to prevent source tokens from attending to previously generated target tokens, leaving the interleaved token order and position IDs untouched. If BLEU rises to the level of batch-streaming (no re-encoding), the paper's diagnosis is confirmed; if it stays at the interleaved level, the gain was due to token reordering, not to the input-attention mask.","tokens_in":22285,"feed_emoji":"⚡","tokens_out":10438,"duration_ms":74158,"temperature":0.7,"pith_summary":"Large language models are built for batch processing, and the standard way to make them stream—re-encoding previously generated outputs with each new input—is computationally heavy. This paper tries to show that almost all of the streaming performance gap comes from a single cause: input tokens are allowed to attend to previously generated output tokens, which never happens in batch mode. The paper argues that the two other suspected mismatches—output tokens' limited attention and broken position IDs—have negligible effects, so re-encoding is largely a waste. It proposes group position encoding, which assigns source and target tokens independent position numberings and uses an attention mask to block source-to-target attention, and shows that fine-tuning an existing instruct LLM this way matches or beats specialized streaming architectures on translation and speech recognition while running up to 11x faster.","feed_headline":"One attention fix streams LLMs 11x faster","feed_subtitle":"Paper shows re-encoding outputs is unnecessary; group position encoding plus a mask makes any LLM stream.","key_machinery":"The central mechanism is the group position encoding paradigm built on the rotary position embedding (RoPE) relative-distance identity. With RoPE, an attention score between a target token and a source token depends on their relative distance n-i = phi + j - i, where $\\varphi$ is the offset between the first target and first source position. Group position encoding fixes phi as a constant instead of letting it grow with source length, so within-source and within-target relative distances match batch mode and cross-segment distances stay stable; the model learns the constant offset during fine-tuning, which is why the choice of phi barely matters. The companion piece is the attention mask that blocks source tokens from attending to generated target tokens, which removes the input-attention mismatch without re-encoding.","core_discovery":"The paper identifies three mismatches between batch and streaming processing of LLMs—input-attention, output-attention, and position-ID—and, through stepwise ablations on IWSLT-17 translation with wait-k policies, finds that only the input-attention mismatch produces substantial BLEU loss (up to 2.20 points). Re-encoding position IDs or KV caches adds at most 0.28 BLEU. A follow-up position-encoding analysis shows that removing absolute positions costs little while removing relative order within source or target costs over 10%, leading to the group position encoding paradigm: source tokens keep positions 0 to M-1, target tokens start at a fixed offset phi, and the attention mask prevents source tokens from seeing target tokens. With LoRA fine-tuning on Gemma2, Phi3, and LLaMA3.1, this method outperforms decoder-only streaming baselines (SimulMask, DST) on translation and (CAAT, wav2vec-S) on ASR, generalizes to batch mode without loss, and needs no architectural changes.","pith_inferences":["A clean control the paper does not run: keep the interleaved token order but block source-to-target attention, and check whether BLEU matches batch-streaming (no re.).","If absolute positions matter as little in other streaming domains as they do here, position handling in real-time LLM applications could be simplified to a constant or omitted entirely.","The phi=0 result suggests source and target can share position IDs, implying even 'position-free' streaming schemes may work for decoder-only LLMs.","The same recipe could carry over to streaming video or sensor inputs, which the paper leaves as future work; the ASR result makes this a concrete next test."],"forward_implications":["A streaming LLM can be produced by LoRA fine-tuning an existing instruct model with a modified attention mask and grouped position IDs, with no architectural changes and no re-encoding.","Inference throughput rises by up to 11.3x under wait-5 and 5.9x under wait-9 in the paper's measurements, because the KV cache is never recomputed.","The same fine-tuned model serves batch processing with no quality loss, so one deployment covers both modes.","The recipe transfers across models (Gemma2, Phi3, LLaMA3.1) and from text translation to speech recognition.","The analysis redirects streaming-LLM design effort from position-ID corrections and KV re-encoding toward attention masking."],"supporting_citations":[{"why":"Defines the wait-k reading/writing policy used in every streaming experiment.","marker":"(Ma et al., 2019)"},{"why":"Provides the rotary position embedding whose relative-distance identity the group position encoding exploits.","marker":"(Su et al., 2024)"},{"why":"Baseline (SimulMask) and a representative of the common assumption that position handling requires special masking.","marker":"(Raffel et al., 2024)"},{"why":"Decoder-only streaming transformer baseline (DST) that the paper's method outperforms.","marker":"(Guo et al., 2024a)"},{"why":"Evidence that decoder-only LLMs learn implicit positional information, used to explain why group positions avoid source-target confusion.","marker":"(Haviv et al., 2022)"},{"why":"IWSLT-17 dataset for the cross-lingual translation evaluations.","marker":"(Cettolo et al., 2017)"},{"why":"LibriSpeech dataset for the automatic speech recognition evaluations.","marker":"(Panayotov et al., 2015)"}],"fun_headline_variants":["One attention fix makes any LLM stream efficiently","Group position encoding closes streaming-batch gap in LLMs","Only input-attention mismatch matters for LLM streaming","Re-encoding outputs is unnecessary: fix input attention only","Stream LLMs faster by preserving relative positions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central diagnosis rests on assuming that the BLEU gain from switching to batch-streaming is caused by blocking input-to-target attention, rather than by the simultaneous reordering of tokens so all source tokens precede all target tokens.","fun_headline_variants_meta":{"raw":{"variants":["One attention fix makes any LLM stream efficiently","Group position encoding closes streaming-batch gap in LLMs","Only input-attention mismatch matters for LLM streaming","Re-encoding outputs is unnecessary: fix input attention only","Stream LLMs faster by preserving relative positions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1357,"prompt_tokens":985,"completion_tokens":372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":297}},"tokens_in":601,"tokens_out":372,"duration_ms":2553,"temperature":1.0,"reasoning_tokens":297,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:52:27.199068+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the interleaved-streaming setting but modify only the attention mask to prevent source tokens from attending to previously generated target tokens, leaving the interleaved token order and position IDs untouched. If BLEU rises to the level of batch-streaming (no re-encoding), the paper's diagnosis is confirmed; if it stays at the interleaved level, the gain was due to token reordering, not to the input-attention mask.","supporting_citations":[],"review_version":1}