Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read A simultaneous speech translation system built from offline Whisper with the AlignAtt policy beats the shared-task baseline by 2 to 22 BLEU across four language pairs, and introduces a new ASR latency metric.

desk verdict Solid engineering and a useful new latency metric, but the En-Ja primary candidate misses the paper's own high-latency cutoff, undercutting the 'all four pairs' claim. read the letter →

arxiv 2506.17077 v1 pith:BXDSBBVV submitted 2025-06-20 cs.CL

classification cs.CL
keywords simultaneousspeechtranslationWhisperAlignAttcascadedEuroLLMASRlatencymetricLevenshteinalignmentvoiceactivitydetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper describes a simultaneous speech translation system that repurposes offline models for streaming use: Whisper large-v3 supplies speech recognition and direct translation, and EuroLLM handles cascade translation into German, Chinese, and Japanese. The authors argue that with the AlignAtt stopping policy, prompting, context, and beam search, these offline models can meet real-time latency regimes while beating the shared-task baseline by 2 BLEU on Czech-to-English and 13-22 BLEU on English-to-German, Chinese, and Japanese. The paper also proposes a new ASR latency metric, based on continuous Levenshtein alignment, intended to measure perceived word-level delay more faithfully than earlier methods. A sympathetic reader would care because the result points to a practical path: simultaneous interpreting quality can track the rapid improvement of offline models without training a dedicated streaming architecture.

What carries the argument

The central mechanism is AlignAtt, a simultaneous policy that inspects the attention weight of the decoder's most-attended source frame and stops generation once that frame falls behind a threshold, letting an otherwise offline model emit translations incrementally. Around that core, the system layers the Whisper large-v3 offline speech model, voice-activity detection with Silero VAD, a speech buffer with length-based trimming, the LocalAgreement prefix-confirmation policy for the EuroLLM text-to-text stage, and the new continuous Levenshtein alignment metric for ASR latency. The AlignAtt policy is what converts offline Whisper into a streaming decoder; LocalAgreement performs the same conversion for EuroLLM without needing attention weights; the latency metric is what lets the authors filter candidates by SLAAL constraints.

What would settle it

Take two streaming ASR systems: one that emits correct words late and one that emits early but wrong words, and compute the proposed metric for both. If human listeners judge the late-but-correct output as more delayed than the early-but-wrong one while the metric orders them the other way, the metric's validity fails. A simpler check: construct a transcript where the aligned gold and ASR words are ambiguous (e.g., repeated words) and verify whether the continuous-alignment tie-break picks the alignment that matches known emission times rather than an arbitrary contiguous segment.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a strong offline speech-to-text model (Whisper large-v3) and a strong text-to-text LLM (EuroLLM-9B-Instruct) can be made to operate in a simultaneous mode with only policy-level additions, and that this combination outperforms the task organizers' dedicated simultaneous baselines on all four language pairs. The direct Czech-to-English system uses Whisper with the AlignAtt attention-threshold policy, beam search, domain prompting, and accumulated context; the cascade systems use Whisper for English ASR and EuroLLM with the LocalAgreement prefix-confirmation policy for translation. The reported gains are 3.3 BLEU (low latency) and 2.2 BLEU (high latency) over the Czech-to-English baseline, and roughly 13, 22, and 18 BLEU over the English-to-German, Chinese, and Japanese baselines, respectively. The paper further claims that its new 'ASR Latency with Continuous Levenshtein Alignment' metric gives a more accurate measure of streaming recognition delay.

Load-bearing premise

The whole latency story rests on the new ASR latency metric: if its continuous-alignment preference does not reflect how humans perceive delay, the candidate filtering and the latency-regime claims are unsupported.

Editorial extensions

If this is right

  • The Czech-to-English system meets the shared task's 2-second low-latency and 4-second high-latency regimes; the English-to-German, Chinese, and Japanese systems meet the high-latency regime.
  • Offline models with an attention-threshold policy can deliver simultaneous translation quality competitive with, and in these tests better than, dedicated online systems.
  • Prompting with in-domain terminology yields a small but consistent ChrF gain on the parliament-style Czech subset.
  • The proposed ASR latency metric can be applied to any streaming ASR system whose words carry emission times, not just this pipeline.
  • Cascading Whisper ASR with EuroLLM translation gives the largest gains on the languages where the direct baseline hallucinates, especially Chinese and Japanese.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct corollary the authors leave implicit is that future gains in offline speech and LLM models could transfer almost automatically to simultaneous translation through policy layers, so specialized streaming training may become less necessary.
  • The new latency metric's parameter choices—especially preferring continuous Copy/Substitute runs—could be stress-tested by comparing its rankings to human judgments of perceived delay on contrived transcripts.
  • Because EuroLLM hallucinated with long contexts and the sentence-trimming strategy failed for Chinese and Japanese, segment-based trimming is likely the safer default for multilingual unbounded translation.
  • A testable extension would apply the continuous Levenshtein latency metric to other streaming ASR systems to see whether it changes published latency comparisons.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes Charles University's IWSLT 2025 simultaneous speech translation submission. The system uses Whisper large-v3 with the AlignAtt policy for Czech-to-English direct translation and for English ASR, followed by EuroLLM-9B with the LocalAgreement policy for cascaded English-to-German, English-to-Chinese, and English-to-Japanese translation. The authors report BLEU/ChrF gains over the organizers' baselines on development data, cover two latency regimes ('low' below 2000 ms and 'high' below 4000 ms SLAAL), and introduce a new ASR latency metric based on continuous character-level Levenshtein alignment. The paper also releases the SimulStreaming implementation and the ASR latency code.

Significance. The submission has concrete practical value: it makes available an open-source simultaneous translation system, applies strong existing policies (AlignAtt, LocalAgreement) to Whisper and EuroLLM, and reports substantial improvements over the official baseline on all four language pairs. If the latency-regime claims survive scrutiny, the paper provides a useful system description and a starting point for reproducible simultaneous speech translation. The proposed ASR latency metric is a potentially useful contribution, but it is currently the main correctness risk because it is unvalidated and is used for candidate selection. The reported gains are also not accompanied by significance tests, so the quantitative claims should be read as development-set observations rather than confirmed test-set results.

major comments (3)
  1. [Section 6.3, Table 9] The primary English-to-Japanese candidate (ASR #22, chunk 2, context 200) is reported with SLAAL = 4654 ms, which exceeds the high-latency threshold of 4000 ms defined in Section 6.1 and the '4-second' regime stated in the introduction. The same table explicitly flags the English-to-Chinese contrastive system (SLAAL = 5449 ms) as not meeting the 4000 ms limit, so the criterion is meant to be enforced. As written, the English-to-Japanese system does not operate in the required high-latency regime, and the abstract's claim of 13-22 BLEU improvements across English-to-German, Chinese, and Japanese loses one language pair unless another candidate is provided or the threshold is clarified. Please reconcile the definitions ('4-5 seconds' in the introduction and Table 1 versus 'below 4000 ms' in Section 6.1) and verify that every primary candidate satisfies the official task limit.
  2. [Section 5.1] The novel ASR latency measure is load-bearing: it is used to filter ASR candidates in Section 6.2 and to check the latency regimes in Section 6.3. However, it is not validated against any existing latency metric, human judgments, or the official task metric. The preference for continuous Copy/Substitute runs is justified only by a single illustrative example, and the tie-breaking among equal-cost alignments is not specified. Since an invalid or biased metric could change the candidate selection and the latency-regime claims, please provide a validation experiment (e.g., correlation with a standard word-level latency measure or with SLAAL on a common benchmark) and specify the algorithm's behavior when multiple optimal alignments exist.
  3. [Sections 5 and 6] All system parameters (MinChunkSize, Frames, Beams, BufferLength, MaxContextLength, prompt and context settings) are tuned on the same development sets used for the final comparisons, and the reported BLEU/ChrF numbers are single runs without variance estimates or significance tests. The baseline comparison in Table 6 is also made on a different segmentation of the same dev data rather than on a held-out test set. The central 'outperform by 2-22 BLEU' claim should therefore be presented as development-set evidence, with an explicit statement of this limitation; where possible, provide confidence intervals or a held-out split for candidate selection versus final reporting.
minor comments (4)
  1. [Section 6.1, Table 2] The text says that experiments with Beams 4 and 8 were run, but Table 2 lists Beams 1, 2, 6, and 5; please correct the text or the table.
  2. [Section 5.1] The latency metric would be considerably easier to reproduce if the dynamic-programming recurrence, the continuous-run preference, and the word-latency aggregation were given as pseudocode or equations rather than only as a prose description.
  3. [Table 8] SLAAL values are reported as '< 0' for hallucinating configurations; since SLAAL is a latency measure, please explain how hallucinations can produce negative values and how such documents are handled in the reported average.
  4. [References] The Radford et al. entry appears as 'Alec Radford and 1 others'; please provide the full author list.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the BLEU/ChrF claims are empirical comparisons against the organizers' external baseline, and reused in-house components are prior work rather than fitted inputs.

full rationale

The paper's central claims are system-building results, not derivations. Improvements over the IWSLT organizers' baseline (2 BLEU on Czech-to-English, 13-22 BLEU on English-to-German/Chinese/Japanese) are measured with standard metrics on the task's dev sets against an external baseline, so no reported score reduces to a parameter fitted for that score. The reused in-house components (Whisper-Streaming, LocalAgreement, the claim that ChrF is more reliable than BLEU for simultaneous translation) are cited as prior engineering or measurement choices; even if those citations were removed, the empirical comparison to the baseline would remain meaningful. The newly proposed ASR latency algorithm is presented as a measurement procedure, not fitted to the authors' outputs, and its use in candidate filtering is an application rather than a derivation of the claimed BLEU gains. The visible internal inconsistency in Table 9, where the English-to-Japanese primary candidate has SLAAL 4654 ms while the paper elsewhere defines the high-latency ceiling as below 4000 ms, is a consistency/correctness concern, not a circularity. No self-definitional step, fitted-input-called-prediction pattern, uniqueness theorem imported from the authors, or ansatz smuggled in via citation is present.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The central claims rest on the reliability of the new latency metric, the adequacy of the dev sets, and the effectiveness of previously published online policies. No new theoretical entities are introduced.

free parameters (8)
  • MinChunkSize = 1.2-1.8 s (Cs-En), 0.05-2.0 s (ASR), 1-3 words (EuroLLM)
    Grid-searched on dev sets; controls how often the system updates.
  • Frames = 4-35
    AlignAtt threshold for attended frame position; tuned via grid search.
  • Beams = 1-8
    Beam search width for Whisper; chosen per system (2 for Cs-En, 1 for ASR).
  • BufferLength = 15-30 s
    Controls how much audio is retained before shifting context; selected 30 s.
  • MaxContextLength = 0-1000 tokens
    Limits prompt/context length for Whisper and EuroLLM; tuned.
  • Prompt text = "This is Chamber of Deputies." or "He starts."
    Hand-selected after experiments to inject domain terminology.
  • StaticPrompt flag = static or non-static
    Whether prompt is pushed away with context; chosen per domain.
  • BufferTrimmingStrategy = Sentences or Segments
    Selected per language to avoid hallucinations in EuroLLM.
assumptions (6)
  • domain assumption AlignAtt policy works effectively with Whisper's decoder attention for streaming.
    Relied on from Wang et al. (2024) and Papi et al. (2023); no new validation in this work.
  • domain assumption LocalAgreement policy works for EuroLLM without attention weights.
    Based on Polak et al. (2022, 2023); used because CTranslate2 does not expose attention.
  • domain assumption Whisper large-v3 and EuroLLM-9B are high-quality backbones for speech and text translation.
    Taken from prior evaluations; the paper does not re-benchmark them independently.
  • domain assumption The IWSLT development sets are representative of the test distribution, and the merged ParCzech dev set is appropriate for unbounded speech.
    The authors merge segments to approximate unsegmented test speech; this may not fully match the test conditions.
  • ad hoc to paper The novel ASR latency metric is a valid measure of perceived latency.
    Proposed in Section 5.1 without external validation or comparison against established latency measures.
  • domain assumption Silero VAD reliably detects voice activity for segmentation.
    Used as a black-box component; default parameters are assumed adequate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025." pith.science (2026). https://pith.science/paper/BXDSBBVV

@misc{pith2026250617077,
  author       = {Pith},
  title        = {Pith review of: Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BXDSBBVV}},
  note         = {Machine review of arXiv:2506.17077}
}
read the original abstract

This paper describes Charles University submission to the Simultaneous Speech Translation Task of the IWSLT 2025. We cover all four language pairs with a direct or cascade approach. The backbone of our systems is the offline Whisper speech model, which we use for both translation and transcription in simultaneous mode with the state-of-the-art simultaneous policy AlignAtt. We further improve the performance by prompting to inject in-domain terminology, and we accommodate context. Our cascaded systems further use EuroLLM for unbounded simultaneous translation. Compared to the Organizers' baseline, our systems improve by 2 BLEU points on Czech to English and 13-22 BLEU points on English to German, Chinese and Japanese on the development sets. Additionally, we also propose a new enhanced measure of speech recognition latency.

Figures

Figures reproduced from arXiv: 2506.17077 by the authors.

Figure 1
Figure 1. Illustration of interrupted vs. continuous align [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Piloting Planetarium Visualizations with LLMs during Live Events in Science Centers

    cs.HC 2026-01 conditional novelty 5.0 of 10

    LLM-based planetarium pilots are unreliable replacements for human pilots but show promise as co-pilots for reducing workload and multitasking in live shows.

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    Idris Abdulmumin, Victor Agostinelli, Tanel Alumäe, Antonios Anastasopoulos, Ashwin, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Fethi Bougares, Roldano Cattoni, Mauro Cettolo, Lizhong Chen, William Chen, Raj Dabre, Yannick Estève, Marcello Federico, Marco Gaido, Dávid Javorský, Marek Kasztelnik, and 30 others. 2025. Findings of the iwslt 2025 eval...

  2. [2]

    Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021. Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22(1)

  3. [3]

    Dominik Mach \'a c ek, Ond r ej Bojar, and Raj Dabre. 2023 a . https://doi.org/10.18653/v1/2023.iwslt-1.12 MT metrics correlate with human ratings of simultaneous speech translation . In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 169--179, Toronto, Canada (in-person and online). Association for Comp...

  4. [4]

    Dominik Mach \'a c ek, Raj Dabre, and Ond r ej Bojar. 2023 b . https://doi.org/10.18653/v1/2023.ijcnlp-demo.3 Turning whisper into real-time transcription system . In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: Sys...

  5. [5]

    Guerreiro, Ricardo Rei, Duarte M

    Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G.C. de Souza , Alexandra Birch, and André F.T. Martins. 2025. https://doi.org/10.1016/j.procs.2025.02.260 Eurollm: Multilingual language models for euro...

  6. [6]

    Sara Papi, Marco Gaido, Matteo Negri, and Luisa Bentivogli. 2024. StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand

  7. [7]

    Sara Papi, Peter Polák, Dominik Mach \'a c ek, and Ond r ej Bojar. 2025. How “real” is your real-time simultaneous speech-to-text translation system? Transactions of the Association for Computational Linguistics, 13:281--313

  8. [8]

    Sara Papi, Marco Turchi, and Matteo Negri. 2023. https://doi.org/10.21437/Interspeech.2023-170 AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation . In Proc. INTERSPEECH 2023

Show all 15 references
  1. [9]

    Peter Polák, Ngoc-Quan Pham, Tuan Nam Nguyen, Danni Liu, Carlos Mullov, Jan Niehues, Ond r ej Bojar, and Alexander Waibel. 2022. https://doi.org/10.18653/v1/2022.iwslt-1.24 CUNI - KIT system for simultaneous speech translation task at IWSLT 2022 . In Proceedings of the 19th In...

  2. [10]

    Peter Polák, Brian Yan, Shinji Watanabe, Alex Waibel, and Ondřej Bojar. 2023. https://doi.org/10.21437/Interspeech.2023-2225 Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoff . In Proc. INTERSPEECH 2023, pages 3979--3983

  3. [11]

    Alec Radford and 1 others. 2022. https://arxiv.org/abs/2212.04356 Robust speech recognition via large-scale weak supervision . Preprint, arXiv:2212.04356

  4. [12]

    Elizabeth Salesky, Kareem Darwish, Mohamed Al-Badrashiny, Mona Diab, and Jan Niehues. 2023. https://doi.org/10.18653/v1/2023.iwslt-1.2 Evaluating multilingual speech translation under realistic conditions with resegmentation and terminology . In Proceedings of the 20th Interna...

  5. [13]

    Lo \"i c Barrault Seamless Communication and 1 others. 2023. Seamless: Multilingual expressive and streaming speech translation

  6. [14]

    Silero Team. 2024. Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier. https://github.com/snakers4/silero-vad

  7. [15]

    Haoyu Wang, Guoqiang Hu, Guodong Lin, Wei-Qiang Zhang, and Jian Li. 2024. https://doi.org/10.21437/Interspeech.2024-1814 Simul-whisper: Attention-guided streaming whisper with truncation detection . In Interspeech 2024, pages 4483--4487

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.