REVIEW 2 major objections 5 minor 27 references
GMU Systems for the IWSLT 2025 Low-Resource Speech Translation Shared Task
T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Fine-tuning a speech-translation model's encoder on in-domain ASR data before end-to-end training improves low-resource speech translation for languages the base model was never trained to recognize, by roughly 1 to 5 BLEU.
desk verdict Useful system report with a transparent appendix, but the ASR-init claim is confounded by a learning-rate change and needs an LR-controlled ablation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is parameter initialization from a fine-tuned ASR encoder. Formally, SeamlessM4T-v2's speech encoder $\theta_{se}$ is fine-tuned on ASR pairs $(x_{sp}, x_{text})$, producing $\theta^{ASR}_{se}$; the E2E ST model's speech encoder is then set to $\theta^{ASR}_{se}$ before training on ST pairs $(x_{sp}, y)$ with the loss $-|y|^{-1} \log p(y|x_{sp};\theta_{se},\theta_{td})$. This transfers the encoder's in-language acoustic and semantic representations into the ST model. A secondary mechanism is multi-task fine-tuning with ST, MT, and knowledge-distillation losses, where a frozen MT teacher trained on the target language provides the distillation signal.
What would settle it
Run a controlled comparison on a language outside SeamlessM4T-v2's ASR coverage, training E2E ST with and without ASR-encoder initialization in the official codebase, with the same hyperparameters and data. The claim would be falsified if the initialized model fails to beat the uninitialized model on the held-out test set by roughly the reported margin, or if the margin only appears in the transformer-toolkit codebase, not the official one.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that the speech encoder is the bottleneck when adapting SeamlessM4T-v2 to languages absent from its ASR training, and that this bottleneck can be removed by component initialization. The procedure is: fine-tune SeamlessM4T-v2 on in-domain ASR data for the target language; then initialize the speech encoder of the ST model with those ASR-trained encoder weights; then fine-tune end-to-end on the ST data. In the authors' internal dev evaluations, this yields +5.1 BLEU for bho-hin, +2.7 for aeb-eng, +0.8 for bem-eng, and a smaller gain for que-spa compared with direct E2E fine-tuning in the same codebase, while for languages SeamlessM4T-v2 already supports in ASR, the initialization has at most a minor negative effect. The paper further claims that direct E2E fine-tuning is strong by itself, and that multi-task training with MT and knowledge-distillation objectives gives small gains only when the fine-tuned MT model beats the E2E ST model.
Load-bearing premise
The load-bearing premise is that differences between the two codebases do not create the reported gains: the ASR-initialization comparison is made inside one reimplementation, while several zero-shot and primary baselines come from the official script, so if the two implementations propagate initialization effects differently, the 1-5 BLEU improvements could be tooling artifacts rather than a property of the ASR encoder.
Editorial extensions
If this is right
- For a new low-resource language, the recipe requires only a few hours of in-domain ASR data plus ST data; no architectural change or new objective is needed.
- For languages the base model already supports in ASR, direct E2E fine-tuning is the best choice; ASR initialization adds little and can slightly hurt.
- Cascaded ASR-to-MT systems remain competitive but generally underperform initialized E2E systems on these language pairs, except for Estonian.
- Multi-task training with MT and knowledge-distillation objectives is worth trying only when the fine-tuned MT model is clearly stronger than the E2E model; otherwise it does not help.
- Component initialization is selective: the ASR encoder helps for unseen languages, while initializing the text decoder from MT helps little, because the target languages are already high-resource.
Reading between the lines
- If the ASR-initialization effect transfers to other large multilingual ST models, the two-stage recipe (ASR fine-tune, then initialize and ST fine-tune) could become a default adaptation protocol for new languages, without any new loss terms.
- The reported gains are measured within one codebase; a natural extension would be to re-run the aeb, bem, bho, and que comparisons in the official codebase to verify the effect is not a tooling artifact.
- Because the paper finds the initialization helps most when the base model lacks ASR coverage, it implies the bottleneck is the speech encoder's language-specific representation, not the decoder; one testable consequence is that adapting the encoder with far fewer ASR hours might retain most of the gain.
- The synthetic-data results for Quechua suggest that when in-domain data are scarce, synthetic augmentation and ASR initialization may interact; this could be tested by ablating the synthetic data while keeping the initialization fixed.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper describes GMU's submissions to the IWSLT 2025 low-resource speech translation shared task. The authors fine-tune SeamlessM4T-v2-Large for nine of the ten language pairs and compare direct E2E ST fine-tuning, cascaded ASR+MT systems, multi-task training with MT and KD objectives, and initialization of ST model components from fine-tuned ASR and/or MT models. Results are reported on internal dev sets, public test sets, and official evaluations. The main claims are that direct E2E fine-tuning is strong, that initializing the ST encoder with a fine-tuned ASR encoder improves ST on languages SeamlessM4T-v2 was not trained on, and that multi-task training can be slightly helpful. The paper includes a code release and an appendix reconciling discrepancies between the HuggingFace and official fine-tuning codebases.
Significance. This is a useful system-description paper with a practical recipe for low-resource ST. Its strengths include reproducible code, a detailed appendix that identifies and partially resolves codebase discrepancies, and a broad exploration across several low-resource language pairs and training paradigms. The potential finding about ASR-encoder initialization for languages outside the base model's ASR coverage is interesting and actionable. However, the current evidence is weakened by a confounded learning-rate comparison in the key table and by the absence of any variance or significance estimates for the reported BLEU differences. If the learning-rate confound is resolved, the paper would provide a clear, falsifiable recipe for future shared-task participants.
major comments (2)
- [Section 4.2 and Table 6] Section 4.2 states that plain E2E ST fine-tuning uses a learning rate of 1e-4, while E2E ST fine-tuning with ASR/MT initialization uses 6e-5. In Table 6, the central within-HF pairs (HF-E2E vs HF-E2E-ASRinit) for aeb (+2.75 dev BLEU), bem (+0.82), and bho (+5.12) therefore vary two factors at once: initialization and learning rate. No direct-E2E run at 6e-5 is reported for these languages, so the observed gains could be a smaller-learning-rate effect on low-resource training rather than a property of the ASR encoder. Please add direct-E2E results at 6e-5 for the same languages, and ideally also ASR-init runs at 1e-4, so that the causal factor behind the claimed improvement can be identified.
- [Tables 2-6 and Appendix B] All reported numbers are single-run point estimates without confidence intervals, error bars, or significance tests. Several conclusions rest on small margins: the bem ASR-init gain is 0.82 BLEU, the que gain is 0.68 BLEU, and the multi-task gains are about 0.7-2 BLEU. Without an estimate of run-to-run variance, it is hard to tell whether these differences are stable or within experimental noise. Please provide multiple seeds for at least the key comparisons (for example, the ASR-init pairs and the multi-task pairs) or otherwise justify that the observed margins exceed run-to-run variability.
minor comments (5)
- [Table 7] The row label '+dropout' appears twice with different BLEU scores (23.50/20.12 and 22.58/19.52). Please clarify which combination of lm_head, dropout, and language-code loss is used in each row.
- [Table 1] The Maltese corpus is labeled 'IWST2025'; this should read 'IWSLT2025'.
- [Appendix B, Table 10] The formatting of the row '+Zevallos et al. (2022) 14.38+Ortega et al. (2020)' is ambiguous. Please split into separate rows and clearly report the cumulative BLEU after each data addition.
- [Section 5.3 and Appendix A] Cross-codebase comparisons in Table 6, such as OFF-E2E versus HF-E2E for the same language, should be marked as preliminary because Appendix A reconciles only direct E2E on aeb. This caveat does not affect the within-HF ASR-init comparisons, which are the basis of the main claim, but it does limit the strength of the 'official codebase yields stronger performance' statement.
- [Section 4.2] The loss-weight choices (alpha=1, beta=1, gamma=2) and the per-language learning-rate and epoch adjustments are described as tuned on the development set. Reporting the tuning range or a small sensitivity analysis would increase confidence that the final settings are not overfit to the dev set.
Circularity Check
No circular derivation: ST results are empirical, measured on held-out and official evaluations; self-citations are non-load-bearing and the learning-rate confound is an experimental validity issue, not a definitional loop.
full rationale
The paper is an empirical system description rather than a deductive derivation. The central claims—direct E2E ST fine-tuning is strong, ASR-encoder initialization helps on languages absent from SeamlessM4T-v2, and multi-task training is mildly helpful—are supported by Table 6 comparisons of independently trained models scored with SacreBLEU on dev and official IWSLT evaluation sets. The ASR-init versus E2E contrast is not true by construction: the former begins from a separately fine-tuned ASR encoder and uses a different learning rate (6e-5 vs 1e-4, Sec. 4.2), which is an experimental confound rather than a circular reduction. The NLLB/Seamless comparisons and the Appendix A codebase reconciliation provide external controls, and no equation in the paper defines one reported quantity from another. Self-citations to shared-task findings and the BIG-C dataset are non-load-bearing data or task references, and no uniqueness theorem or ansatz is imported from the authors' prior work. Accordingly, no circular step is exhibited.
Assumptions & free parameters
free parameters (4)
- Multi-task loss weights alpha, beta, gamma =
alpha=1, beta=1, gamma=2
- Quechua learning rate and maximum epochs =
learning rate 1e-5, max 200 epochs
- Learning rate for initialized models =
6e-5 versus 1e-4
- Batch sizes for speech and text tasks =
120 utterances for speech input, 256 sentences for text input, 72 for est and que ASR
assumptions (4)
- domain assumption SeamlessM4T-v2-Large is a suitable base model whose components can be transferred to languages it has not seen
- domain assumption BLEU, WER, and CER, computed on lowercased and punctuation-stripped text, are valid proxies for translation and recognition quality
- domain assumption The HuggingFace codebase is faithful enough to the official codebase for the compared strategies
- domain assumption External and synthetic datasets, including Azure-synthesized Irish speech and NLLB-mined or model-translated Quechua text, are of sufficient quality for training
Cite this review
Pith. "Pith review of GMU Systems for the IWSLT 2025 Low-Resource Speech Translation Shared Task." pith.science (2026). https://pith.science/paper/HLT4L27Z
@misc{pith2026250521781,
author = {Pith},
title = {Pith review of: GMU Systems for the IWSLT 2025 Low-Resource Speech Translation Shared Task},
year = {2026},
howpublished = {\url{https://pith.science/paper/HLT4L27Z}},
note = {Machine review of arXiv:2505.21781}
}
read the original abstract
This paper describes the GMU systems for the IWSLT 2025 low-resource speech translation shared task. We trained systems for all language pairs, except for Levantine Arabic. We fine-tuned SeamlessM4T-v2 for automatic speech recognition (ASR), machine translation (MT), and end-to-end speech translation (E2E ST). The ASR and MT models are also used to form cascaded ST systems. Additionally, we explored various training paradigms for E2E ST fine-tuning, including direct E2E fine-tuning, multi-task training, and parameter initialization using components from fine-tuned ASR and/or MT models. Our results show that (1) direct E2E fine-tuning yields strong results; (2) initializing with a fine-tuned ASR encoder improves ST performance on languages SeamlessM4T-v2 has not been trained on; (3) multi-task training can be slightly helpful.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Idris Abdulmumin, Victor Agostinelli, Tanel Alumäe, Antonios Anastasopoulos, Ashwin, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Fethi Bougares, Roldano Cattoni, Mauro Cettolo, Lizhong Chen, William Chen, Raj Dabre, Yannick Estève, Marcello Federico, Marco Gaido, Dávid Javorský, Marek Kasztelnik, and 30 others. 2025. Findings of the iwslt 2025 evaluatio...
work page 2025
-
[4]
Milind Agarwal, Sweta Agrawal, Antonios Anastasopoulos, Luisa Bentivogli, Ond r ej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Mingda Chen, William Chen, Khalid Choukri, Alexandra Chronopoulou, Anna Currey, Thierry Declerck, Qianqian Dong, Kevin Duh, Yannick Est \`e ve, Marcello Federico, and 43 others. 2023. https://doi.org/10.18...
-
[5]
Ibrahim Said Ahmad, Antonios Anastasopoulos, Ond r ej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, William Chen, Qianqian Dong, Marcello Federico, Barry Haddow, D \'a vid Javorsk \'y , Mateusz Krubi \'n ski, Tsz Kin Lam, Xutai Ma, Prashant Mathur, Evgeny Matusov, Chandresh Maurya, John McCrae, and 25 others. 2024. https://doi.org/1...
-
[6]
Antonios Anastasopoulos, Lo \"i c Barrault, Luisa Bentivogli, Marcely Zanon Boito, Ond r ej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad, Clara Emmanuel, Yannick Est \`e ve, Marcello Federico, Christian Federmann, Souhir Gahbiche, Hongyu Gong, Roman Grundkiewicz, Barry Haddow, Benjamin Hsu, and 24 others. 2022. https://doi....
-
[7]
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. https://aclanthology.org/2020.lrec-1.520/ Common voice: A massively-multilingual speech corpus . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4218--4222, Marse...
2020
-
[8]
Ronald Cardenas, Rodolfo Zevallos, Reynaldo Baquerizo, and Luis Camacho. 2018. Siminchik: A speech corpus for preservation of southern quechua. ISI-NLP 2, page 21
work page 2018
Show all 27 references
-
[9]
Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2021. https://doi.org/10.21437/Interspeech.2021-329 Unsupervised Cross-Lingual Representation Learning for Speech Recognition . In Interspeech 2021, pages 2426--2430
2021 doi
-
[10]
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020. https://a...
2020 arXiv
-
[11]
Fei He, Shan-Hui Cathy Chu, Oddur Kjartansson, Clara Rivera, Anna Katanova, Alexander Gutkin, Isin Demirsahin, Cibu Johny, Martin Jansche, Supheakmungkol Sarin, and Knot Pipatsrisawat. 2020. https://aclanthology.org/2020.lrec-1.800/ Open-source multi-speaker speech corpora for...
2020
-
[12]
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, ...
2024 doi
-
[13]
Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L. Forcada. 2018. https://doi.org/10.18653/v1/W18-6453 Findings of the WMT 2018 shared task on parallel corpus filtering . In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 726--...
2018 doi
-
[14]
Phuong-Hang Le, Hongyu Gong, Changhan Wang, Juan Pino, Benjamin Lecouteux, and Didier Schwab. 2023. Pre-training for speech translation: CTC meets optimal transport . In International Conference on Machine Learning, pages 18667--18685. PMLR
2023
-
[15]
Ilya Loshchilov and Frank Hutter. 2019. https://arxiv.org/abs/1711.05101 Decoupled Weight Decay Regularization . Preprint, arXiv:1711.05101
2019 arXiv
-
[16]
Yasmin Moslem. 2024. https://arxiv.org/abs/2406.17363 Leveraging synthetic audio data for end-to-end low-resource speech translation . In Proceedings of the 2024 International Conference on Spoken Language Translation (IWSLT 2024), Bangkok, Thailand
2024 arXiv
-
[17]
NLLB Team , Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia-Gonzalez, Pran...
2022
-
[18]
John E Ortega, Richard Castro Mamani, and Kyunghyun Cho. 2020. Neural machine translation with a polysynthetic low resource language. Machine Translation, 34(4):325--346
2020
-
[19]
Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Brussels, Belgium. Association for Computational Linguistics
2018 doi
-
[20]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. https://proceedings.mlr.press/v202/radford23a.html Robust Speech Recognition via Large-Scale Weak Supervision . In Proceedings of the 40th International Conference on Machine Lear...
2023
-
[21]
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021. https://doi.org/10.18653/v1/2021.acl-long.507 CCM atrix: Mining billions of high-quality parallel sentences on the web . In Proceedings of the 59th Annual Meeting of the Associ...
2021 doi
-
[22]
Seamless Communication , Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sa...
2023 arXiv
-
[23]
Seamless Communication , Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei L...
2023 arXiv
-
[24]
Claytone Sikasote, Eunice Mukonde, Md Mahfuz Ibn Alam, and Antonios Anastasopoulos. 2023. https://doi.org/10.18653/v1/2023.acl-long.115 BIG - C : a multimodal multi-purpose dataset for B emba . In Proceedings of the 61st Annual Meeting of the Association for Computational Ling...
2023 doi
-
[25]
J \"o rg Tiedemann. 2012. https://aclanthology.org/L12-1246/ Parallel data, tools and interfaces in OPUS . In Proceedings of the Eighth International Conference on Language Resources and Evaluation ( LREC `12) , pages 2214--2218, Istanbul, Turkey. European Language Resources A...
2012
-
[26]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and ...
2020 doi
-
[27]
Rodolfo Zevallos, Luis Camacho, and Nelsi Melgarejo. 2022. Huqariq: A multilingual speech corpus of native languages of peru forspeech recognition. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5029--5034
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.