REVIEW 3 major objections 4 minor 35 references
HITSZ's End-To-End Speech Translation Systems Combining Sequence-to-Sequence Auto Speech Recognition Model and Indic Large Language Model for IWSLT 2025 in Indic Track
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read An end-to-end system pairing Whisper's encoder with an Indic LLM reaches competitive BLEU across six English–Indic directions.
desk verdict Headline BLEU average is wrong (27.88, not 28.88), but the system description is honest and useful after a fix. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Window-Level Query Transformer (Q-Former), a connector that compresses Whisper's audio features into embeddings Krutrim can read, combined with LoRA low-rank adapters on the LLM's weights. Only these light modules are trained, keeping the 7-billion-parameter Krutrim decoder and Whisper encoder mostly frozen; for Indic-to-English the Whisper encoder is also updated for one epoch. This modularity is what lets a low-resource task inherit Whisper's automatic speech recognition robustness and Krutrim's Indic language knowledge without training a full speech-translation model from scratch.
What would settle it
Run the paper's training recipe on the official IWSLT 2025 Indic track train and development data, decode the test sets, and score the six directions with a standard BLEU scorer against the official references; the central claim is supported only if the per-direction scores match Table 2 and the quoted averages. The quickest internal check is that the English-to-Indic table values average to 27.88, not the abstract's stated 28.88.
Extended reading notes
Core claim
The claim is empirical: on the IWSLT 2025 test sets, the per-direction BLEU scores are 27.00 for English-to-Bengali, 33.84 for English-to-Hindi, 22.81 for English-to-Tamil, 25.02 for Bengali-to-English, 39.29 for Hindi-to-English, and 19.27 for Tamil-to-English, giving the reported averages of 28.88 and 27.86. This establishes that Whisper-large-v2 plus Krutrim-1-instruct, adapted only through a Q-Former and LoRA (with one epoch of Whisper unfreezing for Indic-to-English), is a workable end-to-end route in a low-resource Indic setting. On the development set, parsing only the Chain-of-Thought outputs that follow the required format shows large BLEU improvements, up to 13.84 points for Tamil-to-English, while the overall parse success rate sits near 60–71 percent, so the CoT benefit is real but only partially accessible.
Load-bearing premise
The reported BLEU scores are honest estimates of generalization, which requires that neither Whisper nor Krutrim encountered the IWSLT 2025 test audio or reference translations during their pretraining.
Editorial extensions
If this is right
- If the reported scores hold, an unconstrained Indic speech-translation submission can be built by adapting frozen Whisper and Krutrim weights, without training a model from scratch and without using supplementary speech corpora.
- The two-stage training strategy for English-to-Indic directions, training on short audio clips first to allow larger batches and then incrementally fine-tuning on long clips, is a practical recipe for memory-limited training.
- Chain-of-Thought fine-tuning only pays off when the generated output can be parsed into the final translation, so improving instruction-following and format adherence is the paper's stated next step.
- Per-direction BLEU values range widely, from 19.27 for Tamil-to-English to 39.29 for Hindi-to-English, so the direction averages should be read alongside the language-pair-specific results.
- The system deliberately excludes the IndicVoices corpus to reduce the risk of test-set overlap, meaning the reported results do not claim gains from extra in-domain speech data.
Reading between the lines
- A decisive test of whether Whisper's encoder or Krutrim's decoder limits Indic-to-English quality would be to translate gold transcripts with Krutrim alone and compare the resulting BLEU against the speech-to-text scores; the large gap between Hindi-to-English at 39.29 and Tamil-to-English at 19.27 suggests the encoder is the prime suspect.
- Because parsable Chain-of-Thought outputs carry substantially higher BLEU, techniques that enforce the output format, such as constrained decoding or parser-in-the-loop training, could convert most of the latent gain into usable gain without new data.
- The reported English-to-Indic average of 28.88 does not match the mean of the three per-direction test scores listed in Table 2, which is 27.88; a reproduction should resolve this discrepancy before comparing systems.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports the HITSZ submission to the IWSLT 2025 Indic track for bidirectional speech translation between English and Hindi, Bengali, and Tamil. The proposed end-to-end system uses a frozen Whisper-large-v2 speech encoder and a Krutrim-1-instruct LLM decoder, connected by a Window-Level Q-Former and adapted with LoRA. Models are trained independently per direction. The paper reports average BLEU scores of 28.88 for English-to-Indic and 27.86 for Indic-to-English, and investigates chain-of-thought fine-tuning, reporting large BLEU gains on outputs that can be automatically parsed into the expected format.
Significance. If the results are accurate, the paper demonstrates that a Whisper encoder plus an Indic-specific LLM with Q-Former and LoRA can form a competitive end-to-end speech translation system for low-resource Indic language pairs. The system description is clear and the authors are careful to exclude the IndicVoices corpus due to possible test overlap. However, the paper is a system report without code or checkable artifacts, and the two central empirical claims need correction or clarification: the headline English-to-Indic average is not reproducible from Table 2, and the chain-of-thought gains are measured on a selected subset of outputs. The contribution is moderate and the architecture is largely inherited from SALMONN/Dhwani, but the empirical results could still be useful for the shared task and future system building.
major comments (3)
- [Abstract and Table 2] The abstract's claim of an average BLEU of 28.88 for English-to-Indic directions is not supported by Table 2. The three per-direction test scores (27.00, 33.84, 22.81) have a macro average of 27.88, not 28.88. Notably, the Indic-to-English average in the abstract (27.86) matches the macro average of Table 3 exactly (25.02, 39.29, 19.27), which makes the 28.88 appear to be either a typo or an undocumented segment-weighted micro-average. The paper must either correct the abstract to 27.88 or provide per-direction sentence counts and explain how the weighted average was computed, since the current text does not allow the reader to verify the headline number.
- [Section 5.2 and Table 4] The chain-of-thought improvements in Table 4 are computed only on outputs that successfully parse into the expected format, with parse success rates of 60.43%–71.00% across directions. Because 29–40% of outputs are excluded, the reported gains (e.g., 13.84 BLEU for Tamil-to-English) are not estimates of the CoT method's effect on overall translation quality; the unparsable outputs could be very poor translations, so the overall BLEU might improve less, not at all, or even decrease. The paper should report BLEU computed over all outputs (e.g., by treating unparsable outputs as incorrect or by extracting translations with a fallback parser) or explicitly rephrase the conclusion to state that the gains are conditional on parsing success and cannot be used to claim overall improvement.
- [Section 5.1 and 5.2] The training settings are described asymmetrically. For English-to-Indic, the LoRA rank and alpha, learning-rate schedule, and short/long audio training strategy are specified in detail. For Indic-to-English, the paper only says the setup 'largely mirrored' the English-to-Indic configuration and that WhisperSE was trainable for the first epoch, without restating the LoRA hyperparameters and Q-Former training details. Since the two groups of directions use different procedures, the missing hyperparameters make the Indic-to-English experiments difficult to reproduce or compare. Please state explicitly which hyperparameters were shared and which differed.
minor comments (4)
- [Section 1 and Table 1] Table 1 reports speech hours but not the number of audio segments or sentences per split. Adding sentence counts would not only help readers understand the data scale but also allow verification of any sentence-weighted BLEU averages.
- [Section 5, test-set contamination] The paper excludes IndicVoices because of possible overlap with the test set, but does not discuss whether the pretraining corpora of Whisper-large-v2 or Krutrim-1-instruct might contain the test audio or reference translations. A brief statement on this risk would be appropriate, since the absolute BLEU numbers depend on the assumption that there is no such contamination.
- [Abstract and Section 1] There is a minor grammatical error in the abstract: 'an end-to-end system integrating' should read 'an end-to-end system that integrates' or similar. Also, Figure 1's caption says 'Overview of the our end-to-end system', which should be corrected.
- [Section 5.2, Table 4] Table 4's column header 'Direction CoT Parsing Success Rate BLEU Score' is visually ambiguous. It would be clearer to split the header into separate rows or use a multi-level header.
Circularity Check
No circularity: the paper's central claims are direct empirical evaluations of an adapted Whisper+Krutrim system against external IWSLT data, with no derivation step that reduces to its own inputs.
full rationale
The paper's central claim is an empirical result: an end-to-end speech translation system built from pre-trained Whisper-large-v2 and Krutrim-1-instruct, adapted with a Q-Former and LoRA, and evaluated with BLEU on IWSLT 2025 test sets. This is a measurement, not a derivation, and the test data and reference translations are external to the system's training procedure. The architecture choices follow published prior work (Dhwani, SALMONN, NICT's IWSLT 2024 submission) rather than an unverified self-citation chain; the authors do not invoke a uniqueness theorem or define the target metric in terms of a fitted parameter. The CoT analysis is reported on the subset of responses that parse successfully, and the paper explicitly acknowledges that the model 'did not consistently adhere to the required CoT output format,' so the 13.84 BLEU improvement is presented as a conditional, best-case observation rather than a system-level prediction; this is a selection-bias/validity concern, not a circularity of the kind where an output equals an input by construction. The discrepancy between the abstract's 28.88 average and the 27.88 average implied by Table 2 is a reporting/arithmetic error, not a circular argument. No load-bearing step in the paper reduces, by the paper's own equations or by self-citation, to its own inputs, so the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- LoRA rank r =
8
- LoRA alpha =
32
- Peak and minimum learning rates =
3e-5 and 1e-5
- Short or long audio threshold =
400 characters of English transcription
assumptions (4)
- domain assumption The official IWSLT train, dev, and test split is correctly aligned and free of transcript or translation noise.
- domain assumption Pre-trained Whisper and Krutrim were not trained on the IWSLT 2025 test set.
- domain assumption BLEU is a valid automatic metric for comparing these translation outputs.
- ad hoc to paper The subset of chain-of-thought outputs that parse successfully is representative of all outputs.
Cite this review
Pith. "Pith review of HITSZ's End-To-End Speech Translation Systems Combining Sequence-to-Sequence Auto Speech Recognition Model and Indic Large Language Model for IWSLT 2025 in Indic Track." pith.science (2026). https://pith.science/paper/MQLQKMUL
@misc{pith2026250719616,
author = {Pith},
title = {Pith review of: HITSZ's End-To-End Speech Translation Systems Combining Sequence-to-Sequence Auto Speech Recognition Model and Indic Large Language Model for IWSLT 2025 in Indic Track},
year = {2026},
howpublished = {\url{https://pith.science/paper/MQLQKMUL}},
note = {Machine review of arXiv:2507.19616}
}
abstract
This paper presents HITSZ's submission for the IWSLT 2025 Indic track, focusing on speech-to-text translation (ST) for English-to-Indic and Indic-to-English language pairs. To enhance translation quality in this low-resource scenario, we propose an end-to-end system integrating the pre-trained Whisper automated speech recognition (ASR) model with Krutrim, an Indic-specialized large language model (LLM). Experimental results demonstrate that our end-to-end system achieved average BLEU scores of $28.88$ for English-to-Indic directions and $27.86$ for Indic-to-English directions. Furthermore, we investigated the Chain-of-Thought (CoT) method. While this method showed potential for significant translation quality improvements on successfully parsed outputs (e.g. a $13.84$ BLEU increase for Tamil-to-English), we observed challenges in ensuring the model consistently adheres to the required CoT output format.
Figures
Reference graph
Works this paper leans on
-
[1]
Milind Agarwal, Sweta Agarwal, Antonios Anastasopoulos, Luisa Bentivogli, Ond r ej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Mingda Chen, and 1 others. 2023. Findings of the iwslt 2023 evaluation campaign. Association for Computational Linguistics
work page 2023
-
[2]
Ibrahim Said Ahmad, Antonios Anastasopoulos, Ond r ej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, William Chen, Qianqian Dong, Marcello Federico, Barry Haddow, D \'a vid Javorsk \'y , Mateusz Krubi \'n ski, Tsz Kin Lam, Xutai Ma, Prashant Mathur, Evgeny Matusov, Chandresh Maurya, John McCrae, and 25 others. 2024. https://doi.org/1...
-
[3]
Ashkan Alinejad and Anoop Sarkar. 2020. Effectively pretraining a speech translation decoder with machine translation data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8014--8020
work page 2020
-
[4]
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. https://doi.org/10.48550/arXiv.2006.11477 wav2vec 2.0: a framework for self-supervised learning of speech representations . arXiv preprint. ArXiv:2006.11477 [cs]
-
[5]
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2019. Pre-training on high-resource speech recognition improves low-resource speech-to-text translation. In 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 58--68. Association for Computational Linguistics
work page 2019
-
[6]
Alexandre B \'e rard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018. End-to-end automatic speech translation of audiobooks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6224--6228. IEEE
2018
-
[7]
Alexandre B \'e rard, Olivier Pietquin, Laurent Besacier, and Christophe Servan. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text translation. In NIPS Workshop on end-to-end learning for speech and audio processing
2016
-
[8]
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. 2023. https://proceedings.mlr.press/v202/chen23ag.html BEATs : audio pre-training with acoustic tokenizers . In Proceedings of the 40th International Conference on Machine Learning , pages 5178--5193. PMLR. ISSN: 2640-3498 shortConferenceName: ICML
work page 2023
Show all 35 references
-
[9]
Gonzalez, Ion Stoica, and Eric P
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\
2023
-
[10]
Raj Dabre and Haiyue Song. 2024. https://doi.org/10.18653/v1/2024.iwslt-1.3 NICT `s cascaded and end-to-end speech translation systems using whisper and I ndic T rans2 for the I ndic task . In Proceedings of the 21st International Conference on Spoken Language Translation (IWS...
2024 doi
-
[11]
Chitale, Raghavan Ak, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M
Jay Gala, Pranjal A. Chitale, Raghavan Ak, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. https://doi.org/10.48550/arXiv.2305.16307 In...
-
[12]
Alex Graves, Santiago Fern \'a ndez, Faustino Gomez, and J \"u rgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369--376
2006
-
[13]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations
2022
-
[14]
Tahir Javed, Janki Atul Nawale, Eldho Ittan George, Sakshi Joshi, Kaushal Santosh Bhogale, Deovrat Mehendale, Ishvinder Virender Sethi, Aparna Ananthanarayanan, Hafsah Faquih, Pratiti Palit, Sneha Ravishankar, Saranya Sukumaran, Tripura Panchagnula, Sunjay Murali, Kunal Sharad...
2024 arXiv
-
[15]
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, page 6282. Association...
2020
- [16]
-
[17]
Bo Li, Shaolin Zhu, and Lijie Wen. 2025. https://aclanthology.org/2025.coling-main.346/ MIT -10 M : A large scale parallel corpus of multilingual image translation . In Proceedings of the 31st International Conference on Computational Linguistics, pages 5154--5167, Abu Dhabi, ...
2025
-
[18]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. https://proceedings.mlr.press/v202/li23q.html BLIP -2: bootstrapping language-image pre-training with frozen image encoders and large language models . In Proceedings of the 40th International Conference on Machine L...
2023
-
[19]
Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2020. Bridging the modality gap for speech-to-text translation. arXiv preprint arXiv:2010.14920
2020 arXiv
-
[20]
Benjamin Marie, Haipeng Sun, Rui Wang, Kehai Chen, Atsushi Fujita, Masao Utiyama, and Eiichiro Sumita. 2019. https://doi.org/10.18653/v1/W19-5330 NICT ' s unsupervised neural and statistical machine translation systems for the WMT 19 news translation task . In Proceedings of t...
2019 doi
-
[21]
Lambert Mathias and William Byrne. 2006. Statistical phrase-based speech translation. In 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, volume 1, pages I--I. IEEE
2006
-
[22]
Hermann Ney. 1999. Speech translation: Coupling of recognition and translation. In 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No. 99CH36258), volume 1, pages 517--520. IEEE
1999
-
[23]
Gabriel Peyr \'e , Marco Cuturi, and 1 others. 2019. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning , 11(5-6):355--607
2019
-
[24]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. https://proceedings.mlr.press/v202/radford23a.html Robust speech recognition via large-scale weak supervision . In Proceedings of the 40th International Conference on Machine Lear...
2023
-
[25]
Sanket Shah, Kavya Ranjan Saxena, Kancharana Manideep Bharadwaj, Sharath Adavanne, and Nagaraj Adiga. 2025. IndicST : Indian multilingual translation corpus for evaluating speech large language models. In Proc. ICASSP
2025
-
[26]
Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2020. https://doi.org/10.18653/v1/2020.acl-main.324 Knowledge distillation for multilingual unsupervised neural machine translation . In Proceedings of the 58th Annual Meeting of the Associatio...
2020 doi
-
[27]
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. 2024. https://openreview.net/forum?id=14rn7HpKVk SALMONN : Towards generic hearing abilities for large language models . In The Twelfth International Conference on Learning ...
2024
-
[28]
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prang...
2022 arXiv
-
[29]
Gállego, José A
Ioannis Tsiamas, Gerard I. Gállego, José A. R. Fonollosa, and Marta R. Costa-jussà. 2024. https://doi.org/10.18653/v1/2024.findings-acl.847 Pushing the limits of zero-shot end-to-end speech translation . In Findings of the Association for Computational Linguistics : ACL 2024 ,...
2024 doi
-
[30]
Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020. Fairseq s2t: Fast speech-to-text modeling with fairseq. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International...
2020
-
[31]
Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2017. https://doi.org/10.18653/v1/D17-1155 Instance weighting for neural machine translation domain adaptation . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, page...
2017 doi
-
[32]
Chen Xu, Rong Ye, Qianqian Dong, Chengqi Zhao, Tom Ko, Mingxuan Wang, Tong Xiao, and Jingbo Zhu. 2023. Recent advances in direct speech-to-text translation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 6796--6804
2023
-
[33]
Shaolin Zhu, Menglong Cui, and Deyi Xiong. 2024. https://aclanthology.org/2024.lrec-main.1444/ Towards robust in-context learning for machine translation with large language models . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Langua...
2024
-
[34]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[35]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.