{"id":"1945c4a7-4d07-4c71-b0e6-9e85fbae4e9d","arxiv_id":"2506.17077","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"The CUNI system improves simultaneous translation over the IWSLT baseline on development sets by 2 to 22 BLEU points, and proposes a new ASR latency metric.","lead":"This paper describes Charles University's system for the IWSLT 2025 simultaneous speech translation task, combining the offline Whisper model with the AlignAtt online policy, and using EuroLLM for cascade translation into German, Chinese, and Japanese. It reports large BLEU gains over the organizers' baseline on development sets and introduces a new character-level measure of speech recognition latency.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"En-Ja primary candidate's reported SLAAL (4654 ms) exceeds the paper's own high-latency threshold (<4000 ms), so the claim of covering all four language pairs in the required regime is internally contradicted.","rationale":"I read the paper as a system description whose central claim is that all four CUNI systems beat the organizers' baseline within the IWSLT 2025 latency regimes. I first checked the reader's flagged assumption, the new ASR latency metric. That metric is used to rank ASR candidates, but the final systems are validated with the established SLAAL metric reported in Tables 6 and 9, so a biased ASR metric would not by itself collapse the headline claims. The more acute problem is that one of the reported SLAAL values contradicts the paper's own threshold. Table 9 lists the En-Ja primary candidate at 4654 ms SLAAL, while Section 6.1 defines high latency as below 4000 ms and the introduction says the required regimes are 2 s and 4 s. The EnZh contrastive row with 5449 ms is explicitly called out as not meeting the 4000 ms limit, so the authors enforce that criterion elsewhere. Unless the '4-5 seconds' phrase in the introduction and Table 1 reflects an actual task definition up to 5000 ms, the En-Ja system is outside the regime and the '18 BLEU' Japanese gain is not a valid within-regime comparison. I keep the verdict CONDITIONAL because the issue is localized and fixable: correct the reported number, provide a compliant En-Ja candidate, or clarify the official threshold. The reader's weakest assumption points to a metric that is not the pivotal vulnerability; the reported SLAAL inconsistency is a more direct and falsifiable check.","tokens_in":9851,"tokens_out":15185,"duration_ms":145176,"concrete_test":"Re-run the official StreamLAAL metric on the English-to-Japanese primary configuration from Table 9 (ASR #22, chunk 2, context 200) on the ACL6060 dev set, and check the IWSLT 2025 high-latency criterion. If SLAAL exceeds 4000 ms, the En-Ja system fails the paper's own latency constraint; if the official cutoff is instead 5000 ms, the paper must correct the Section 6.1 statement and the introduction's '4-second' language. Either way, the reported En-Ja SLAAL of 4654 ms and the stated threshold cannot both be true as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most concrete threat to the central claim is an internal inconsistency in the reported latency results, not the novel ASR metric. Section 6.1 defines the IWSLT 2025 high-latency threshold as SLAAL below 4000 ms, and the introduction states the required regimes are 2-second and 4-second. Table 9, introduced as 'High-latency simultaneous translation results,' reports the English-to-Japanese primary candidate (ASR #22, chunk 2, context 200) with SLAAL = 4654 ms, which exceeds that threshold by 654 ms. The same table marks the EnZh contrastive system with SLAAL 5449 ms as not meeting the 4000 ms limit, showing the criterion was meant to be enforced. If 4654 ms is accurate, the English-to-Japanese system does not operate in the high-latency regime defined by the paper, so the claimed '18 BLEU' improvement on Japanese is not a within-regime result and the abstract's '13-22 BLEU' claim loses one language pair. The introduction and Table 1 also describe the high regime as '4-5 seconds,' which conflicts with the '4-second' and '<4000 ms' definitions; the paper must clarify which is the official task threshold and verify that every primary candidate satisfies it.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes Charles University's IWSLT 2025 simultaneous speech translation submission. The system uses Whisper large-v3 with the AlignAtt policy for Czech-to-English direct translation and for English ASR, followed by EuroLLM-9B with the LocalAgreement policy for cascaded English-to-German, English-to-Chinese, and English-to-Japanese translation. The authors report BLEU/ChrF gains over the organizers' baselines on development data, cover two latency regimes ('low' below 2000 ms and 'high' below 4000 ms SLAAL), and introduce a new ASR latency metric based on continuous character-level Levenshtein alignment. The paper also releases the SimulStreaming implementation and the ASR latency code.","tokens_in":10141,"tokens_out":8144,"duration_ms":69802,"significance":"The submission has concrete practical value: it makes available an open-source simultaneous translation system, applies strong existing policies (AlignAtt, LocalAgreement) to Whisper and EuroLLM, and reports substantial improvements over the official baseline on all four language pairs. If the latency-regime claims survive scrutiny, the paper provides a useful system description and a starting point for reproducible simultaneous speech translation. The proposed ASR latency metric is a potentially useful contribution, but it is currently the main correctness risk because it is unvalidated and is used for candidate selection. The reported gains are also not accompanied by significance tests, so the quantitative claims should be read as development-set observations rather than confirmed test-set results.","major_comments":[{"comment":"The primary English-to-Japanese candidate (ASR #22, chunk 2, context 200) is reported with SLAAL = 4654 ms, which exceeds the high-latency threshold of 4000 ms defined in Section 6.1 and the '4-second' regime stated in the introduction. The same table explicitly flags the English-to-Chinese contrastive system (SLAAL = 5449 ms) as not meeting the 4000 ms limit, so the criterion is meant to be enforced. As written, the English-to-Japanese system does not operate in the required high-latency regime, and the abstract's claim of 13-22 BLEU improvements across English-to-German, Chinese, and Japanese loses one language pair unless another candidate is provided or the threshold is clarified. Please reconcile the definitions ('4-5 seconds' in the introduction and Table 1 versus 'below 4000 ms' in Section 6.1) and verify that every primary candidate satisfies the official task limit.","section":"Section 6.3, Table 9"},{"comment":"The novel ASR latency measure is load-bearing: it is used to filter ASR candidates in Section 6.2 and to check the latency regimes in Section 6.3. However, it is not validated against any existing latency metric, human judgments, or the official task metric. The preference for continuous Copy/Substitute runs is justified only by a single illustrative example, and the tie-breaking among equal-cost alignments is not specified. Since an invalid or biased metric could change the candidate selection and the latency-regime claims, please provide a validation experiment (e.g., correlation with a standard word-level latency measure or with SLAAL on a common benchmark) and specify the algorithm's behavior when multiple optimal alignments exist.","section":"Section 5.1"},{"comment":"All system parameters (MinChunkSize, Frames, Beams, BufferLength, MaxContextLength, prompt and context settings) are tuned on the same development sets used for the final comparisons, and the reported BLEU/ChrF numbers are single runs without variance estimates or significance tests. The baseline comparison in Table 6 is also made on a different segmentation of the same dev data rather than on a held-out test set. The central 'outperform by 2-22 BLEU' claim should therefore be presented as development-set evidence, with an explicit statement of this limitation; where possible, provide confidence intervals or a held-out split for candidate selection versus final reporting.","section":"Sections 5 and 6"}],"minor_comments":[{"comment":"The text says that experiments with Beams 4 and 8 were run, but Table 2 lists Beams 1, 2, 6, and 5; please correct the text or the table.","section":"Section 6.1, Table 2"},{"comment":"The latency metric would be considerably easier to reproduce if the dynamic-programming recurrence, the continuous-run preference, and the word-latency aggregation were given as pseudocode or equations rather than only as a prose description.","section":"Section 5.1"},{"comment":"SLAAL values are reported as '< 0' for hallucinating configurations; since SLAAL is a latency measure, please explain how hallucinations can produce negative values and how such documents are handled in the reported average.","section":"Table 8"},{"comment":"The Radford et al. entry appears as 'Alec Radford and 1 others'; please provide the full author list.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop-style system description rather than a full methodology paper, but the internal contradiction on the English-to-Japanese latency is a substantive correctness issue that should be fixed before acceptance. The open-source release and the explicit focus on realistic unbounded speech are genuine strengths; the unvalidated ASR latency metric is the main technical risk to the paper's claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a competent shared-task system description: the authors combine AlignAtt with Whisper for unbounded speech, add a EuroLLM cascade with LocalAgreement, and release the implementation as SimulStreaming. Second, the paper's central claim—that all four language pairs meet the IWSLT 2025 latency regimes—is internally contradicted by its own Table 9. The English-to-Japanese primary candidate (ASR #22, chunk 2, context 200) reports SLAAL 4654 ms, which exceeds the paper's stated high-latency threshold of below 4000 ms. The same table flags an En-Zh contrastive system as not meeting the limit, so this is not a typo in the threshold definition. The introduction also says the high regime is '4-5 seconds,' which conflicts with the 'below 4000ms' used in Section 6.1. The stress-test note hits the load-bearing point: if 4654 ms is accurate, the 18 BLEU gain on Japanese is not a within-regime result, and the abstract's '13-22 BLEU' claim loses a language pair.\n\nWhat is genuinely new: the ASR latency metric based on continuous Levenshtein alignment (Section 5.1). It is a plausible improvement over word-level or interrupted alignment, and the authors publish the code. The engineering combination is also solid—merging Simul-Whisper with Whisper-Streaming, adding beam search and prompting, and handling EuroLLM's lack of attention weights via LocalAgreement are all sensible choices. The empirical work is honest about the dev-set evaluation, but there are no error bars and parameters are tuned on the same sets used for final comparison. That is typical for shared-task papers, but it means the headline BLEU numbers should be read as indicative, not conclusive.\n\nThe internal latency inconsistency is the real problem. It is not a minor issue. The paper needs to clarify the official threshold, verify every primary candidate against it, and either fix the En-Ja candidate or retract the claim that it covers that language pair in the high-latency regime. The lack of test-set validation is a lesser concern, since the test set results presumably appear in the IWSLT findings; still, the dev-set claims should be labeled as such.\n\nWho is this paper for? People building simultaneous speech translation systems will find SimulStreaming and the latency metric useful. It deserves a serious referee, but only with a request for major revision: the En-Ja contradiction must be addressed before the paper's core claims can be trusted.","headline":"Solid engineering and a useful new latency metric, but the En-Ja primary candidate misses the paper's own high-latency cutoff, undercutting the 'all four pairs' claim.","tokens_in":10666,"tokens_out":1607,"would_cite":false,"duration_ms":17217,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A simultaneous speech translation system built from offline Whisper with the AlignAtt policy beats the shared-task baseline by 2 to 22 BLEU across four language pairs, and introduces a new ASR latency metric.","keywords":["simultaneous speech translation","Whisper","AlignAtt","cascaded speech translation","EuroLLM","ASR latency metric","Levenshtein alignment","voice activity detection"],"falsifier":"Take two streaming ASR systems: one that emits correct words late and one that emits early but wrong words, and compute the proposed metric for both. If human listeners judge the late-but-correct output as more delayed than the early-but-wrong one while the metric orders them the other way, the metric's validity fails. A simpler check: construct a transcript where the aligned gold and ASR words are ambiguous (e.g., repeated words) and verify whether the continuous-alignment tie-break picks the alignment that matches known emission times rather than an arbitrary contiguous segment.","tokens_in":9651,"feed_emoji":"🎙️","tokens_out":6932,"duration_ms":63093,"temperature":0.7,"pith_summary":"This paper describes a simultaneous speech translation system that repurposes offline models for streaming use: Whisper large-v3 supplies speech recognition and direct translation, and EuroLLM handles cascade translation into German, Chinese, and Japanese. The authors argue that with the AlignAtt stopping policy, prompting, context, and beam search, these offline models can meet real-time latency regimes while beating the shared-task baseline by 2 BLEU on Czech-to-English and 13-22 BLEU on English-to-German, Chinese, and Japanese. The paper also proposes a new ASR latency metric, based on continuous Levenshtein alignment, intended to measure perceived word-level delay more faithfully than earlier methods. A sympathetic reader would care because the result points to a practical path: simultaneous interpreting quality can track the rapid improvement of offline models without training a dedicated streaming architecture.","feed_headline":"Offline Whisper streams translations beating baselines by up to 22 BLEU","feed_subtitle":"Combining Whisper with the AlignAtt policy improves quality on four language pairs and adds a sharper ASR latency measure.","key_machinery":"The central mechanism is AlignAtt, a simultaneous policy that inspects the attention weight of the decoder's most-attended source frame and stops generation once that frame falls behind a threshold, letting an otherwise offline model emit translations incrementally. Around that core, the system layers the Whisper large-v3 offline speech model, voice-activity detection with Silero VAD, a speech buffer with length-based trimming, the LocalAgreement prefix-confirmation policy for the EuroLLM text-to-text stage, and the new continuous Levenshtein alignment metric for ASR latency. The AlignAtt policy is what converts offline Whisper into a streaming decoder; LocalAgreement performs the same conversion for EuroLLM without needing attention weights; the latency metric is what lets the authors filter candidates by SLAAL constraints.","core_discovery":"On the paper's own terms, the central discovery is that a strong offline speech-to-text model (Whisper large-v3) and a strong text-to-text LLM (EuroLLM-9B-Instruct) can be made to operate in a simultaneous mode with only policy-level additions, and that this combination outperforms the task organizers' dedicated simultaneous baselines on all four language pairs. The direct Czech-to-English system uses Whisper with the AlignAtt attention-threshold policy, beam search, domain prompting, and accumulated context; the cascade systems use Whisper for English ASR and EuroLLM with the LocalAgreement prefix-confirmation policy for translation. The reported gains are 3.3 BLEU (low latency) and 2.2 BLEU (high latency) over the Czech-to-English baseline, and roughly 13, 22, and 18 BLEU over the English-to-German, Chinese, and Japanese baselines, respectively. The paper further claims that its new 'ASR Latency with Continuous Levenshtein Alignment' metric gives a more accurate measure of streaming recognition delay.","pith_inferences":["A direct corollary the authors leave implicit is that future gains in offline speech and LLM models could transfer almost automatically to simultaneous translation through policy layers, so specialized streaming training may become less necessary.","The new latency metric's parameter choices—especially preferring continuous Copy/Substitute runs—could be stress-tested by comparing its rankings to human judgments of perceived delay on contrived transcripts.","Because EuroLLM hallucinated with long contexts and the sentence-trimming strategy failed for Chinese and Japanese, segment-based trimming is likely the safer default for multilingual unbounded translation.","A testable extension would apply the continuous Levenshtein latency metric to other streaming ASR systems to see whether it changes published latency comparisons."],"forward_implications":["The Czech-to-English system meets the shared task's 2-second low-latency and 4-second high-latency regimes; the English-to-German, Chinese, and Japanese systems meet the high-latency regime.","Offline models with an attention-threshold policy can deliver simultaneous translation quality competitive with, and in these tests better than, dedicated online systems.","Prompting with in-domain terminology yields a small but consistent ChrF gain on the parliament-style Czech subset.","The proposed ASR latency metric can be applied to any streaming ASR system whose words carry emission times, not just this pipeline.","Cascading Whisper ASR with EuroLLM translation gives the largest gains on the languages where the direct baseline hallucinates, especially Chinese and Japanese."],"supporting_citations":[{"why":"Whisper backbone: the offline speech model that supplies ASR and direct translation.","marker":"Radford et al. 2022"},{"why":"AlignAtt policy: attention-threshold stopping criterion that enables simultaneous decoding.","marker":"Papi et al. 2023"},{"why":"Simul-Whisper: prior implementation showing AlignAtt works with Whisper; this paper extends it.","marker":"Wang et al. 2024"},{"why":"Whisper-Streaming: prior implementation providing unbounded speech interface and VAD integration.","marker":"Macháček et al. 2023b"},{"why":"LocalAgreement policy: prefix-confirmation mechanism used for EuroLLM cascade translation.","marker":"Polák et al. 2023"},{"why":"EuroLLM: the 9B instruction-tuned LLM used for English-to-German/Chinese/Japanese translation.","marker":"Martins et al. 2025"},{"why":"StreamLAAL: the SLAAL metric used for translation latency constraints.","marker":"Papi et al. 2024"},{"why":"Defines the realistic unbounded simultaneous translation setup and the prototypical pipeline the systems follow.","marker":"Papi et al. 2025"},{"why":"ACL6060: the realistic conditions dev set used for the English-to-German/Chinese/Japanese evaluation.","marker":"Salesky et al. 2023"},{"why":"IWSLT 2025 task definition and the organizers' baseline used for BLEU comparisons.","marker":"Abdulmumin et al. 2025"}],"fun_headline_variants":["Whisper + AlignAtt boosts simultaneous translation by up to 22 BLEU","Offline Whisper streams simultaneous translation, up to 22 BLEU","CUNI's Whisper-LLM combo beats simultaneous baselines by up to 22 BLEU","Streaming Whisper and EuroLLM: simultaneous translation up by 22 BLEU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole latency story rests on the new ASR latency metric: if its continuous-alignment preference does not reflect how humans perceive delay, the candidate filtering and the latency-regime claims are unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Whisper + AlignAtt boosts simultaneous translation by up to 22 BLEU","Offline Whisper streams simultaneous translation, up to 22 BLEU","CUNI's Whisper-LLM combo beats simultaneous baselines by up to 22 BLEU","Streaming Whisper and EuroLLM: simultaneous translation up by 22 BLEU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001222,"raw_usage":{"total_tokens":5014,"prompt_tokens":923,"completion_tokens":4091,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":3995}},"tokens_in":539,"tokens_out":4091,"duration_ms":26429,"temperature":1.0,"reasoning_tokens":3995,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:11:35.835665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two streaming ASR systems: one that emits correct words late and one that emits early but wrong words, and compute the proposed metric for both. If human listeners judge the late-but-correct output as more delayed than the early-but-wrong one while the metric orders them the other way, the metric's validity fails. A simpler check: construct a transcript where the aligned gold and ASR words are ambiguous (e.g., repeated words) and verify whether the continuous-alignment tie-break picks the alignment that matches known emission times rather than an arbitrary contiguous segment.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"StreamLAAL: the SLAAL metric used for translation latency constraints."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the realistic unbounded simultaneous translation setup and the prototypical pipeline the systems follow."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"IWSLT 2025 task definition and the organizers' baseline used for BLEU comparisons."}],"review_version":2}