REVIEW 4 major objections 6 minor 27 references
Instituto de Telecomunica\c{c}\~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A 1.5B speech-to-text model, trained by aligning a speech encoder with a small text decoder, gets English ASR right but fails to translate or answer when prompted.
desk verdict Honest small-scale speech-LM system paper with a genuinely useful pseudo-labeling recipe, but the cost-parity claim overreaches and the paper's own ST/SQA results contradict the multitask story. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a two-stage curriculum. In the modality-alignment stage, only three 1D convolutional pre-encoder layers and a two-layer Conformer-like adapter are trained—the language model stays frozen—on ASR data for a single epoch, optionally receiving a length hint. In the instruction fine-tuning stage every component trains jointly on ASR, speech translation, and spoken QA, with audio representations prepended to text embeddings and task/language tokens such as <|translate|><de> steering generation. The adapter compresses audio in time and projects it into the decoder's embedding space; the paper's argument depends on this alignment being enough for a 1.5B decoder to internalize the task tokens.
What would settle it
On the shared-task test inputs, compute the tag-following rate: for each <|translate|><de|> input, whether the output is actually German rather than an English transcription of the audio. The paper already reports transcribe-instead-of-translate failures; a systematic tag-following rate near chance for the non-ASR tags would settle that the two-stage curriculum does not teach instruction-following for translation and spoken QA.
Extended reading notes
Core claim
On its own terms, the paper shows that one epoch of ASR-only modality alignment followed by joint instruction fine-tuning turns a w2v-BERT 2.0 continuous speech encoder plus a Qwen 2.5 1.5B decoder into a unified speech-to-text model with solid English ASR and weak translation and spoken-QA performance. The model does not reliably obey task and language tags: it repeats words, transcribes where it should translate, and emits the question where it should emit the answer. The paper reads these failures as evidence that the data mixture must be rebalanced so ASR does not dominate and that the curriculum itself needs to be more carefully designed; it names more data, stronger multilingual backbones, and longer audio context as the immediate next steps.
Load-bearing premise
The recipe assumes that a single epoch of ASR-only alignment gives the speech encoder enough grounding that a later mixed instruction-tuning stage can teach a 1.5B decoder to switch between transcribing, translating, and answering; the paper's own ST and SQA results show this grounding suffices for transcription but not for the other two tasks.
Editorial extensions
If this is right
- English ASR at 0.15 WER on technical spontaneous speech means a 1.5B-parameter system with open data is a usable ASR front end for this domain.
- The two-stage recipe does not automatically yield joint multitask competence; translation and spoken QA require a rebalanced mixture, so future systems should not copy the ASR-dominated data split.
- Multi-model oracle pseudolabeling with a COMETKiwi threshold keeps more and more diverse translation training data than any single MT model, easing the data bottleneck for speech translation.
- The 120-second audio cutoff discards much of the spoken QA corpus, so extending context length is a direct, predicted route to better spoken question answering.
Reading between the lines
- A natural next experiment the paper does not run: rebalance the instruction fine-tuning mixture (downsample ASR, upsample speech translation and spoken QA) and measure whether tag-following improves; the paper's own failure analysis predicts it would.
- The same pipeline with a 3B or larger decoder would separate encoder-alignment limits from decoder-capacity limits; the authors hint at this but do not test it.
- The oracle-pseudolabeling pipeline could be ablated to single-model pseudolabeling to see whether diversity or threshold filtering drives the retained-data gain.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes the IT-IST submission to the IWSLT 2025 Instruction Following Short Track. The system couples a pretrained w2v-BERT 2.0 speech encoder with a Qwen 2.5 1.5B (or 0.5B) text decoder and trains in two stages: a modality-alignment stage on ASR data, followed by instruction fine-tuning on ASR, speech translation (ST), and spoken question answering (SQA). Training uses open CC-BY data plus synthetic data filtered with COMETKiwi. The reported official scores (Table 3) are an English ASR WER of 0.15, ST COMET of 0.34 for both language pairs, and SQA BERTScore values between 0.14 and 0.22. The paper's central claim is that a small-scale model can achieve 'similar results at a fraction of the cost of larger LMs.'
Significance. If the efficiency claim were supported, the paper would be a useful data point for building speech-to-text systems with sub-2B backbones, and the release of code and data under permissive licenses would aid reproducibility. The synthetic-data pipeline and the oracle-based COMETKiwi filtering are also potentially reusable. However, the evidence in the paper supports only the English ASR result; the ST and SQA numbers are low, and the authors' own manual inspection reports the model ignoring task tags and repeating words. No baseline against any larger system is provided, so the parity-at-lower-cost claim is unsubstantiated. The paper is honest about its limitations, but the advertised contribution is larger than what the results demonstrate.
major comments (4)
- [§1, Introduction] The first paragraph claims that 'with highly filtered and synthetic data, we can enable similar results at a fraction of the cost of larger LMs.' No comparison to any larger LM appears anywhere in the paper, and Table 3 gives only absolute scores for the submitted system. Without a matched baseline (e.g., the same training recipe with a 7B decoder, or official IWSLT baselines and other systems' scores), this parity claim is unsupported. Either add such a comparison or substantially qualify the claim to, for example, 'competitive ASR at low cost'.
- [§3.2 and Table 3] The two-stage curriculum is presented as the mechanism that 'enables multi-task capabilities,' but the reported ST and SQA results do not support this: COMET is 0.34 for both en–de and en–zh, and SQA BERTScore ranges from 0.14 to 0.22. Section 4 further states that 'at times, the model repeats the same word or ignores the task tag and transcribes the audio segment rather than translating it' and that the model sometimes generates the question itself instead of the answer. These outcomes directly contradict the claimed multi-task benefit. The paper should report a task-tag-following metric or a per-instance error analysis so the reader can see how often the model follows the instruction; without this, the central multi-task claim is not evidenced.
- [§3.2–§3.3 and Table 1] No ablation isolates the effect of the proposed curriculum or data mixture. The pipeline has several components—modality alignment length, single-epoch MA, the 95% length hint, COMETKiwi thresholds, synthetic ST data, synthetic QA data, and unanswerable-question generation—but the paper never varies any of them. A reader cannot tell whether the English ASR score is due to the two-stage design, the specific data mixture, or simply the pretrained encoder. At minimum, an IFT-only versus MA+IFT comparison and a comparison of the final data mixture against a more balanced one are needed to support the paper's stated contribution.
- [§4, Results and §5, Conclusions] The paper reports a single run for each task and gives no confidence intervals, significance tests, or multiple-seed variation. Given that the main claim is about efficiency and that the ST/SQA results are weak, it is important to state that the official numbers are point estimates from a single submission and to note whether any selection among checkpoints or decoding configurations occurred. If these are official single-run scores, the authors should say so explicitly.
minor comments (6)
- [Table 3 and surrounding text] The table header 'en en-de en-zh' with columns 'ASR SQA ST SQA ST SQA' is confusing because it mixes target-language labels with task labels. Please restructure the table so that each task-language pair is a clearly labeled column, and specify that WER is for English ASR only.
- [§3.3, Spoken Question Answering] The text says COMETKiwi 'performed unreliably' for short answers, but it does not explain how the answers were filtered or whether the reported SQA scores reflect answers that passed any quality filter. Please clarify the answer-filtering procedure and its effect on training data size.
- [Figure 1 and §3.1] The figure shows the task and language tags as '<TASK>', '<LANG>', and '<|translate|>', but the text does not specify how these tags are tokenized for Qwen 2.5. Please state whether these are added to the vocabulary or mapped to existing tokens.
- [Table 1] The row 'CV 16.1 PL' is not defined in the table or text. Please spell out what 'PL' means (e.g., pseudo-labeled) and the source of that data.
- [References] In the IWSLT 2025 findings reference, the author list contains the incomplete entry 'Ashwin,' which should be expanded or replaced with the full author list or 'et al.' as appropriate.
- [Limitations] The Limitations section says 'we do not see much of the SQA data' because of the 120-second audio cutoff, but no concrete numbers are given. Reporting the actual number of retained SQA training examples after the cutoff would help readers calibrate how data-limited the SQA result is.
Circularity Check
No significant circularity: the paper's reported results come from the official IWSLT shared-task metrics, not from quantities fitted in its own training pipeline.
full rationale
The derivation chain is a standard two-stage training recipe (modality alignment then instruction fine-tuning) evaluated on the external IWSLT 2025 shared-task metrics. The COMETKiwi thresholds in Section 3.3 are used only to filter and select synthetic training data (pseudolabeled translations and QA pairs); the reported ST and SQA scores are official COMET and BertScore numbers from Abdulmumin et al. (2025), so no fitted input is renamed as a prediction. The paper's main claim ('similar results at a fraction of the cost of larger LMs') is overbroad and not supported by a baseline comparison, and the ST/SQA results are weak by the authors' own admission; however, an unsupported or contradicted claim is a correctness and evidence problem, not circularity. The self-references (Sannigrahi et al., 2024, for prompt design; Martins et al., 2024, and Rei et al., 2022, within the pseudolabeling pipeline) are not load-bearing: the choices are externally testable and do not constitute the target result, and the evaluation is independent of all of them. No equation or construction in the paper reduces a reported result to an input by definition.
Assumptions & free parameters
free parameters (10)
- Modality alignment learning rate =
3e-3
- Beam size =
3
- Repetition penalty =
1.6
- Nucleus sampling temperature =
1.2
- COMETKiwi ST filter threshold =
0.85
- COMETKiwi SQA question filter threshold =
0.80
- Length hint probability =
0.95
- Audio length cutoff =
120 seconds
- MA epochs =
1
- Unanswerable questions per context =
2
assumptions (5)
- domain assumption Pretrained speech encoder and LM can be aligned by training only adapter layers in the first stage while freezing both backbones.
- domain assumption COMETKiwi scores are a valid quality proxy for selecting synthetic ST and QA data.
- domain assumption One epoch of ASR-only alignment suffices to establish a usable speech embedding space for later multitask IFT.
- domain assumption Length hints improve token generation (from Deitke et al., 2024).
- standard math Cross-entropy loss on reference transcripts is the correct training objective for all three tasks.
Cite this review
Pith. "Pith review of Instituto de Telecomunica\c{c}\~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning." pith.science (2026). https://pith.science/paper/SZMFCUQ7
@misc{pith2026250617019,
author = {Pith},
title = {Pith review of: Instituto de Telecomunica\cc\~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/SZMFCUQ7}},
note = {Machine review of arXiv:2506.17019}
}
read the original abstract
This paper presents the IT-IST submission to the IWSLT 2025 Shared Task on Instruction Following Speech Processing. We submit results for the Short Track, i.e., speech recognition, translation, and spoken question answering. Our model is a unified speech-to-text model that integrates a pre-trained continuous speech encoder and text decoder through a first phase of modality alignment and a second phase of instruction fine-tuning. Crucially, we focus on using small-scale language model backbones (< 2B) and restrict to high-quality, CC-BY data along with synthetic data generation to supplement existing resources.
Figures
Reference graph
Works this paper leans on
-
[2]
arXiv preprint arXiv:2503.01743
Phi-4-mini tech- nical report: Compact yet powerful multimodal lan- guage models via mixture-of-loras. arXiv preprint arXiv:2503.01743. Duarte Miguel Alves, José Pombal, Nuno M Guerreiro, Pedro Henrique Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and Andre Martins
-
[3]
arXiv preprint arXiv:2503.10620
From tower to spire: Adding the speech modality to a text-only llm. arXiv preprint arXiv:2503.10620. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gre- gor Weber
-
[6]
arXiv preprint arXiv:2407.10759
Qwen2-audio technical report. arXiv preprint arXiv:2407.10759. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil- iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou
-
[7]
arXiv preprint arXiv:2311.07919
Qwen-audio: Advancing universal audio understanding via unified large-scale audio- language models. arXiv preprint arXiv:2311.07919. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna
-
[8]
In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805
Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805. IEEE. Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi
work page 2022
-
[9]
arXiv preprint arXiv:2409.17146
Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146. Marco Gaido, Sara Papi, Matteo Negri, and Luisa Ben- tivogli
-
[11]
arXiv preprint arXiv:2408.00118
Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others
-
[12]
arXiv preprint arXiv:2407.21783
The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruom- ing Pang
Show all 27 references
-
[14]
In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2024, pages 4552–4572, Miami, Florida, USA
WavLLM: Towards robust and adaptive speech large language model. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2024, pages 4552–4572, Miami, Florida, USA. Association for Computational Linguistics. Zhichao Huang, Rong Ye, Tom Ko, Qianqian Dong, Shanbo C...
2024
-
[15]
arXiv preprint arXiv:2312.13585
Speech translation with large language models: An industrial practice. arXiv preprint arXiv:2312.13585. Alkis Koudounas, Eliana Pastor, Giuseppe Attanasio, Vittorio Mazzia, Manuel Giollo, Thomas Gueudre, Elisa Reale, Luca Cagliero, Sandro Cumani, Luca de Alfaro, and 1 others
-
[17]
In ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 13326–13330
V oxtlm: Unified decoder-only models for consoli- dating speech recognition, synthesis and speech, text continuation tasks. In ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 13326–13330. IEEE. Pedro Henrique Martins,...
2024
-
[18]
Preprint, arXiv:2409.16235
Eurollm: Multilingual language models for europe. Preprint, arXiv:2409.16235. NLLB Team, Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume ...
-
[21]
MLS: A Large-Scale Multilingual Dataset for Speech Re- search. In Proc. Interspeech 2020, pages 2757–2761. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever
2020
-
[22]
In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid)
CometKiwi: IST-unbabel 2022 sub- mission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics. Paul K Rubenstein, Chu...
2022
-
[23]
arXiv preprint arXiv:2306.12925
Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925. Sonal Sannigrahi, Thiago Fraga-Silva, Youssef Oualil, and Christophe Van Gysel
-
[24]
arXiv preprint arXiv:2310.13289
Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289. Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux
-
[25]
Preprint, arXiv:2101.00390
V oxpop- uli: A large-scale multilingual speech corpus for rep- resentation learning, semi-supervised learning and interpretation. Preprint, arXiv:2101.00390. Changhan Wang, Anne Wu, and Juan Pino
-
[26]
arXiv preprint arXiv:2007.10310
Cov- ost 2 and massively multilingual speech-to-text trans- lation. arXiv preprint arXiv:2007.10310. Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shu- jie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei
2007 arXiv
-
[27]
arXiv e-prints, pages arXiv–2305
Viola: Unified codec language models for speech recognition, synthesis, and translation. arXiv e-prints, pages arXiv–2305. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jian- hong ...
2024 arXiv
-
[2015]
In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210
Librispeech: an asr cor- pus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert
2015
-
[2018]
Spoken squad: A study of mit- igating the impact of speech recognition errors on listening comprehension. Proc. Interspeech 2018, pages 3459–3463. Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fang- ming Liu, Xiwen Zhang, Nicholas D Lane, and Meng- wei Xu
2018
-
[2020]
Preprint, arXiv:2005.08100
Conformer: Convolution-augmented transformer for speech recognition. Preprint, arXiv:2005.08100. Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Ling- wei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, and Furu Wei
2005 arXiv
-
[2021]
Preprint, arXiv:2111.09344
The people’s speech: A large-scale diverse english speech recognition dataset for com- mercial usage. Preprint, arXiv:2111.09344. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahr...
-
[2022]
arXiv preprint arXiv:2207.04672
No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur
-
[2023]
arXiv preprint arXiv:2312.05187
Seamless: Mul- tilingual expressive and streaming speech translation. arXiv preprint arXiv:2312.05187. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, and 1 others
-
[2024]
In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21318–21340, Miami, Florida, USA
Twists, humps, and peb- bles: Multilingual speech recognition models exhibit gender performance gaps. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21318–21340, Miami, Florida, USA. Association for Computational Lin- guistics....
2024
-
[2025]
In Proceedings of the 22nd Interna- tional Conference on Spoken Language Translation (IWSLT 2025), Vienna, Austria (in-person and on- line)
Findings of the iwslt 2025 eval- uation campaign. In Proceedings of the 22nd Interna- tional Conference on Spoken Language Translation (IWSLT 2025), Vienna, Austria (in-person and on- line). Association for Computational Linguistics. To appear. Abdelrahman Abouelenin, Atabak A...
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.