{"id":"3c9d70fb-8eff-4556-9f76-c78c097938be","arxiv_id":"2501.00868","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LSG uses an LLM's own distribution difference from a wait-1 baseline to decide read/write timing, and reports strong results on simultaneous translation and streaming ASR.","lead":"This paper introduces LSG, a framework that lets a large language model decide when to read more input and when to write output during simultaneous translation and streaming speech recognition. The model compares its own output distribution on the current input with a low-latency wait-1 baseline and writes when the difference crosses a threshold, with a confidence rule and a latency range limit as safeguards.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The w/o Range ablation (Table 2) shows the KL/confidence readiness signal fails without the hand-set window, so the LLM's policy-making may be a minor modulation of a developer-chosen latency band.","rationale":"The reader's weakest assumption is that the KL divergence is a reliable readiness signal, supported by the dominant role of the range constraint in the w/o Range ablation. I agree that this is the load-bearing concern, and I sharpen it: the ablation suggests not merely that the signal is weak, but that the developer-set window is the effective policy, leaving the LLM only a bounded choice that may be irrelevant. The concrete test of a fixed midpoint policy within the same windows would settle whether the LLM's in-window decisions matter at all. Because the reader already identified this issue and the recommended CONDITIONAL verdict appropriately requires such evidence, no verdict change is needed; the acceptance conditions should explicitly include this fixed-policy comparison or an equivalent analysis of the realized write-offset distribution.","tokens_in":16038,"tokens_out":6257,"duration_ms":59168,"concrete_test":"Run the WMT15 De→En experiments with a fixed 'midpoint' policy that sets g_i = L + i - 1 + floor(U/2) for the same [L,U] grid as Table 5, using the same LoRA-tuned Llama2-7B and greedy decoding. Compare SacreBLEU at matched AL against the LSG rows: if the midpoint policy is within 1.0 BLEU at each matched AL, the KL/confidence decision provides no measurable policy value beyond the hand-set window. Additionally, report the distribution of the realized write offset g_i - (L+i-1); a distribution concentrated at 0 or U indicates the signal is not making graded in-window decisions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the LLM itself decides generation timing via Eq.3 and Eq.4. Table 2's w/o Range row is the most direct evidence against this: when Eq.5's window is removed, BLEU drops to 21.95 at AL 3.62 (vs. 31.60 at AL 4.42 for full LSG) and reaches only 29.90 at AL 12.91 (vs. 33.22 at AL 7.37). The unbounded KL+confidence rule is therefore not a reliable readiness signal: it either writes too early or waits too long. The range constraint (Eq.5) is a hard developer-set latency window [L+i-1, L+i-1+U], and it is doing the policy work; the LLM is only choosing within a narrow, pre-bounded interval. If the distribution-difference signal were truly informative, removing the range constraint should at worst degrade latency control, not collapse quality by roughly 10 BLEU. This makes the 'off-the-shelf LLM as policy-maker' claim unsupported: the actual policy is largely the developer's choice of L and U, with the LLM contributing an unquantified correction. The paper never reports how often or how far the realized write position g_i falls inside the window, so there is no evidence the LLM's decisions differ from a trivial fixed rule within that window.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LSG, a framework in which an off-the-shelf LLM serves as both generator and read/write policy-maker for simultaneous generation. At each step the LLM computes the KL divergence between its next-token distribution given the current source prefix and given the wait-1 baseline prefix; if this divergence exceeds a threshold δ, or if the top probability exceeds α, the LLM writes a token, and the allowed write positions are constrained to a developer-set window [L+i−1, L+i−1+U]. Experiments on simultaneous text translation (WMT15 De→En, MuST-C En→De), simultaneous speech translation, and streaming ASR on CoVoST2 Fr→En claim state-of-the-art latency-quality trade-offs.","tokens_in":16311,"tokens_out":5861,"duration_ms":49529,"significance":"If the claimed behavior holds, LSG would provide a training-free alternative to DP-trained policies for simultaneous generation with LLMs, and the paper's breadth across three tasks is valuable. The paper ships code, includes ablations and a sufficiency-rate analysis, and uses open-source models, which aids reproducibility. However, the specific evidence that the LLM's distribution-comparison signal is doing the policy work is weak; the range constraint appears to carry most of the policy burden, and key operating points are chosen per dataset without sensitivity analysis.","major_comments":[{"comment":"The w/o Range ablation is the most direct evidence against the central claim of LLM-driven policy-making. Without Eq. 5's window, the method yields BLEU 21.95 at AL 3.62, nearly 10 BLEU below the full LSG at AL 4.42, and only reaches BLEU 29.90 even at AL 12.91, which is far worse than the full method at AL 7.37. This indicates that the KL/confidence conditions in Eqs. 3-4 do not by themselves identify reliable write positions; the hand-set window supplies the actual policy. To support the claim that the LLM decides timing, the paper should report, within the window, how often and how far the realized write positions depart from a trivial rule (e.g., always write at the lower bound or upper bound), and compare against a random-in-window baseline.","section":"Ablation Study, Table 2"},{"comment":"δ and α are set per dataset/direction (e.g., δ=9.0, α=0.6 for De→En; δ=7.0, α=0.5 for Fr→En) with no sensitivity analysis. Because the policy is an empirical heuristic, the reported latency-quality curves may reflect tuned operating points. Please provide results over a grid of δ and α at a fixed [L,U], or show that the method is robust to these thresholds, so the reader can assess whether the comparisons are fair.","section":"System Settings and Appendix Tables 5-7"},{"comment":"The streaming ASR table lists two unlabeled LSG rows (AL 3161.25, WER 31.71; AL 4342.23, WER 23.76) without stating which [L,U] and thresholds they correspond to. The baselines are reported at a single AL of 5684.38, so the claim of 'comparable quality with a delay of about 3 seconds' is supported by only one point and the higher-quality LSG point is at a latency below the baselines' latency. Please provide a full latency-WER curve with labeled operating points and have the baselines evaluated at multiple latencies where possible.","section":"Main Results, Table 1"},{"comment":"The enforcement mechanism of the range constraint is not specified. It is unclear whether the system is forced to write when j reaches the upper bound L+i−1+U, or whether the constraint only limits when the KL/confidence conditions are consulted. This is load-bearing because the policy's behavior at the boundary determines the realized latency and quality. Please state the algorithm explicitly (e.g., pseudocode) for a single step, including what happens if neither Eq. 3 nor Eq. 4 is satisfied at the upper bound.","section":"Range constraint, Eq. 5"},{"comment":"The motivating claim that the KL divergence between current and wait-1 distributions is a reliable and monotone readiness signal rests on a single illustrative example. Provide aggregate evidence over the test set: for example, the distribution of the divergence at the true aligned read position versus one step earlier, or the correlation between the divergence and the latency penalty of writing at that position. Without such evidence, the choice of δ remains arbitrary and the failure mode seen in the w/o Range ablation is unsurprising.","section":"Introduction, Figure 1"}],"minor_comments":[{"comment":"The paper calls the method 'off-the-shelf' but for text tasks fine-tunes Llama2-7B-chat with LoRA on 100k samples; please clarify that the framework is applicable to off-the-shelf models but the reported text experiments use fine-tuned adapters.","section":"System Settings / Experiment Details"},{"comment":"The segment-size ablation shows two rows for 320 ms and 640 ms but only one row for 960 ms; report the low-latency operating point for 960 ms as well, or explain why it is absent.","section":"Table 3"},{"comment":"The legend uses 'Diseg' while the text uses 'DiSeg'; please ensure consistent naming.","section":"Figure 3(c)"},{"comment":"The BLEU differences in Figure 3 are not accompanied by significance tests or confidence intervals; since greedy decoding is deterministic, a bootstrap over test sentences would strengthen the claim that the improvements are not due to test-set noise.","section":"Evaluation"},{"comment":"The sufficiency rate depends on eflomal alignments; alignment errors could bias the comparison, so please consider reporting agreement with a second aligner or providing alignment-quality statistics.","section":"Quality of Policy"}],"recommendation":"major_revision","confidential_remarks":"The authors are known contributors to simultaneous translation and the code release is a plus. The main worry is that the paper's framing overstates the role of the LLM as 'policy-maker' when the developer-set window appears to be the effective policy; this is fixable with additional analyses but requires real work, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: The paper has a genuinely new mechanism and strong empirical breadth, but the central 'LLM as policy-maker' claim is not actually supported by its own ablation. The w/o Range row in Table 2 is exactly the evidence that matters: unbounded KL+confidence writes too early (AL 3.62, BLEU 21.95) or waits too long (AL 12.91, BLEU 29.90), while the full system with the hand-set window [L, L+U] gets the reported trade-offs. So the developer's L and U are setting the latency-quality curve; the LLM is choosing within a narrow band. The paper never shows how often the LLM's choice differs from a trivial fixed rule inside that window, which would be the evidence needed to substantiate the policy-maker framing.\n\nWhat is new and good: the specific readiness signal, comparing the LLM's next-token distribution under the current prefix vs a wait-1 baseline, is not in the cited LLM-based methods (Simul-LLM, TransLLaMa, Agent-SiMT), which use external or fixed policies. The experiments cover text and speech simultaneous translation plus streaming ASR, and the method beats strong baselines. The ablations are honest, including the w/o Range row that undercuts the headline claim. Code is public.\n\nSoft spots, in order: (1) The load-bearing range constraint has no sensitivity analysis; delta, alpha, L, U are per-dataset choices, and there are no error bars or significance tests anywhere. (2) Simul-LLM and TransLLaMa are cited but not compared against; for an LLM-based method that is a gap. (3) Off-the-shelf is inaccurate for the text experiments, which LoRA-fine-tune Llama2-7B-chat on 100k samples; the speech experiments use an already-instruction-tuned Qwen-Audio. (4) The streaming ASR table (Table 1) reports two LSG rows without a clear statement of what differs, presumably two latency settings, and needs a footnote.\n\nBottom line: The recipe works and is potentially useful for real-time systems. But the paper's own data say the LLM is modulating a developer-chosen latency window, not making the policy. A referee can fix this with a modest revision: reframe the contribution as a constrained policy with LLM-based within-window selection, add a sensitivity analysis and a distribution of write positions inside the window, and run the missing baselines. I would send it to peer review, not desk reject, and I would cite it as a baseline in future work, albeit with a critical note. For a reading group, it is a nice case study in how an ablation can contradict the abstract.","headline":"A novel, useful simultaneous-generation recipe whose headline claim is undercut by its own ablation: the developer-set range window, not the LLM, is doing the policy work.","tokens_in":16884,"tokens_out":3899,"would_cite":true,"duration_ms":32143,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an off-the-shelf large language model can act as both the policy-maker and the generator in simultaneous generation, deciding on its own when to read more input and when to write output, and that this beats…","keywords":["simultaneous generation","large language models","read/write policy","simultaneous machine translation","streaming automatic speech recognition","wait-k policy","KL divergence","latency-quality trade-off"],"falsifier":"Run LSG on a language pair with heavy reordering (for example Japanese\\textrightarrow English) and compare its read/write decisions against oracle aligned-source decisions; if the KL threshold frequently disagrees with the oracle and the confidence condition is what actually triggers writes, the central policy signal is not doing the work. Alternatively, replace the two-distribution comparison with a fixed wait-k schedule using the same $[L,U]$ range constraint and the same LLM; if the fixed schedule matches LSG's quality at every latency, the distribution signal adds nothing.","tokens_in":15811,"feed_emoji":"⚡","tokens_out":9255,"duration_ms":76838,"temperature":0.7,"pith_summary":"The paper sets out to remove the need for complex dynamic-programming policy training in simultaneous generation. It proposes LSG, in which an off-the-shelf large language model both decides when to read more input and when to write output, using the wait-1 policy as a minimum-latency baseline. At each step the LLM compares its next-token distribution on the current streaming prefix with its distribution on the wait-1 prefix; when the KL divergence between these exceeds a threshold $\\delta$, it writes, otherwise it reads. A confidence condition catches cases where the baseline already provides enough information, and a range constraint keeps latency inside a developer-set window. The paper reports state-of-the-art quality-latency trade-offs on simultaneous text translation, simultaneous speech translation, and streaming ASR using open-source LLMs.","feed_headline":"Off-the-shelf LLMs write as they read in live translation and ASR","feed_subtitle":"A single LLM decides each read-or-write step by comparing its next-token distribution to a minimum-latency baseline, and beats specialized…","key_machinery":"The mechanism is a two-distribution comparison made by the same LLM at each step. LSG computes the next-token distribution from the currently available source, $p(y_i|x_{\\le j}, y_{<i})$, and from the wait-1 baseline input, $p(y_i|x_{\\le i}, y_{<i})$, and takes the READ action unless $D_{\\mathrm{KL}}(p(y_i|x_{\\le j}, y_{<i}) \\,\\|\\, p(y_i|x_{\\le i}, y_{<i})) > \\delta$. The confidence condition $\\max p(y_i|x_{\\le j}, y_{<i}) > \\alpha$ handles false negatives where the baseline already supplies enough source information, and the $[L,U]$ range constraint caps how far the policy can stray from a fixed latency window. The wait-1 policy is the chosen baseline because it is the smallest usable input prefix, making it the natural anchor against which extra reading is measured.","core_discovery":"The central discovery is that policy-making for streaming generation can be derived from the LLM's own next-token distributions instead of being learned through training. Given the source prefix $x_{\\le j}$ and previously generated words $y_{<i}$, the LLM produces $p(y_i|x_{\\le j}, y_{<i})$, and LSG writes exactly when $D_{\\mathrm{KL}}(p(y_i|x_{\\le j}, y_{<i}) \\,\\|\\, p(y_i|x_{\\le i}, y_{<i})) > \\delta$. The authors show that this distribution difference grows as more source content arrives and becomes significant once the necessary content is available, while a small remaining difference means the wait-1 prefix is sufficient. The confidence condition $\\max p(y_i|x_{\\le j}, y_{<i}) > \\alpha$ allows a write even when the KL condition is not met, and the range constraint $g_i \\in [\\min(L+i-1,J), \\min(L+i-1+U,J)]$ keeps the policy within controllable latency. The claim is that this combination of two probability tests and one window constraint lets one off-the-shelf LLM act as both policy-maker and generator, achieving state-of-the-art results across three streaming tasks.","pith_inferences":["If the KL divergence between current-prefix and wait-1-prefix distributions is a general signal of source sufficiency, the same two-condition recipe should transfer to other streaming text tasks such as incremental summarization or live code completion, where an explicit alignment signal is unavailable.","The paper's ablation shows the range constraint is the largest single contributor, so the marginal benefit of the KL signal over a fixed wait-k schedule confined to the same $[L,U]$ window is an open question that the paper does not isolate.","A direct test of the policy signal would be to run LSG on a language pair with extreme word-order divergence and check whether the $\\delta$ threshold, rather than the $\\alpha$ confidence rule, triggers the writes that match oracle aligned-source decisions."],"forward_implications":["No dynamic-programming policy learning is needed; the policy is derived from the LLM's own distributions at inference time, and the only training used in the text experiments is a standard LoRA fine-tuning of the generator.","The same LSG recipe transfers across modalities, since it works with Llama2-7B-chat on text-to-text translation and with Qwen-Audio on speech-to-text translation and streaming ASR.","Latency and quality are controlled by the threshold $\\delta$, the confidence $\\alpha$, and the range window $[L,U]$, so an engineer can set a worst-case response delay in advance.","Better LLMs produce better simultaneous generation under LSG, as Bayling-13B outperforms Bayling-7B on Chinese\\textrightarrow English at the same latency levels.","LSG's policies have higher sufficiency than wait-k and HMT policies applied to the same LLM, meaning the model writes more often only after the aligned source information is available."],"supporting_citations":[{"why":"Defines the wait-k policy and Average Lagging latency metric; supplies the wait-1 baseline that LSG's read/write comparison is built on.","marker":"Ma et al. (2019)"},{"why":"Provides Llama2-7B-chat, the off-the-shelf LLM used for simultaneous text translation experiments.","marker":"Touvron et al. (2023)"},{"why":"Provides Qwen-Audio, the speech LLM used for simultaneous speech translation and streaming ASR.","marker":"Chu et al. (2023)"},{"why":"LoRA fine-tuning used to adapt Llama2-7B-chat on 100k translation samples before running LSG.","marker":"Hu et al. (2021)"},{"why":"SimulEval toolkit used to compute Average Lagging and SacreBLEU for all streaming evaluations.","marker":"Ma et al. (2020a)"},{"why":"HMT is the strongest dynamic-programming-trained encoder-decoder baseline; LSG is compared against it and its policy sufficiency is measured on the same LLM.","marker":"Zhang and Feng (2023b)"},{"why":"Agent-SiMT is the prior LLM-based simultaneous translation method that couples an external policy module with the LLM; LSG outperforms it.","marker":"Guo et al. (2024)"},{"why":"CoVoST2 is the dataset used for simultaneous speech translation and streaming ASR experiments.","marker":"Wang, Wu, and Pino (2020)"}],"fun_headline_variants":["LLMs time their own writes during streaming input","One off-the-shelf LLM reads and writes in real time","LLM writes when next-token difference beats threshold","Read/write policy from LLM's own probability checks","Streaming: LLM decides write timing from token distributions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the KL divergence between the LLM's next-token distribution on the current prefix and its distribution on the wait-1 prefix is a reliable, monotone signal that enough source content has been read, so that crossing a fixed threshold $\\delta$ marks the right moment to write.","fun_headline_variants_meta":{"raw":{"variants":["LLMs time their own writes during streaming input","One off-the-shelf LLM reads and writes in real time","LLM writes when next-token difference beats threshold","Read/write policy from LLM's own probability checks","Streaming: LLM decides write timing from token distributions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000901,"raw_usage":{"total_tokens":3901,"prompt_tokens":993,"completion_tokens":2908,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":2828}},"tokens_in":609,"tokens_out":2908,"duration_ms":21893,"temperature":1.0,"reasoning_tokens":2828,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:40:51.342373+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LSG on a language pair with heavy reordering (for example Japanese\\textrightarrow English) and compare its read/write decisions against oracle aligned-source decisions; if the KL threshold frequently disagrees with the oracle and the confidence condition is what actually triggers writes, the central policy signal is not doing the work. Alternatively, replace the two-distribution comparison with a fixed wait-k schedule using the same $[L,U]$ range constraint and the same LLM; if the fixed schedule matches LSG's quality at every latency, the distribution signal adds nothing.","supporting_citations":[],"review_version":1}