Speech Translation Refinement using Large Language Models

Huaixia Dou; Jie Zhu; Junhui Li; Lifan Guo; Xinglin Lyu; Xinyu Tian

REVIEW 3 major objections 6 minor 63 references

Speech Translation Refinement using Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Refining speech-to-text translation and ASR transcription together, in one LLM pass, consistently improves translation quality beyond fixing the translation alone.

desk verdict Joint ASR+ST refinement is a real, reproducible win for LLM-based speech translation post-editing, but missing confidence intervals and in-sample training data keep it from being a clean bill of health. read the letter →

arxiv 2501.15090 v1 pith:ZGF2PU7B submitted 2025-01-25 cs.CL

Huaixia Dou , Xinyu Tian , Xinglin Lyu , Jie Zhu , Junhui Li , Lifan Guo This is my paper · ORCID

classification cs.CL

keywords speechtranslationLLMrefinementjointautomaticrecognitionin-contextlearningparameter-efficientfine-tuningdocument-levelcontextpost-editing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that an LLM should fix both the automatic speech recognition output and the speech translation output at the same time, rather than polishing the translation by itself. On MuST-C and CoVoST 2 covering seven English-to-X tasks, this joint refinement raises translation-quality scores by up to 4.81 BLEU and 0.0755 COMET over the unrefined system and improves transcription WER. The gains appear in both in-context learning with GPT-3.5-turbo and in parameter-efficient fine-tuning of LLaMA3-8B and Mistral-12B, and document-level context adds a further boost. The reason the approach matters is that speech translation errors often stem from source transcription errors, so correcting both lets each output repair the other.

What carries the argument

The load-bearing object is the RefineBoth task and its prompt format: an instruction, optional in-context examples, and a query that asks the LLM to output a refined transcription followed by a refined translation. Fine-tuning uses a two-stage procedure in which the LLM first learns to generate the ASR transcription and ST translation, then learns to refine both, and document-level context is handled by chunk-based decoding that concatenates K neighboring sentences with sentence indices to preserve alignment.

What would settle it

Run the fine-tuned RefineBoth model on test audio with a new domain shift (for example, telephonic noise or an unseen accent) that creates ASR error types absent from the training split while keeping the ST system fixed; if BLEU drops below the translation-only RefineST baseline, the joint-refinement claim collapses. Alternatively, if shuffling sentences within a document produces no BLEU drop, the claimed context benefit is not real.

Watch

Extended reading notes

Core claim

The central discovery is that errors in ASR transcription and ST translation are complementary, not independent, and an LLM can exploit this. The paper defines the joint refinement task (A,S)→(A′,S′), where A is the noisy transcription and S the noisy translation, and the LLM first generates a corrected transcription A′ and then a corrected translation S′. Controlled comparisons show that this RefineBoth task beats RefineST, which only refines the translation, on BLEU across all evaluated language pairs, with differences reported as statistically significant at p<0.01, while COMET is similar or better. The paper further claims that a two-stage fine-tuning strategy — first teaching the LLM to produce A and S, then to refine both — outperforms refinement-only fine-tuning, and that incorporating document-level context of three sentences gives additional gains.

Load-bearing premise

The refinement training data is made by running the same ASR and ST systems on the training set and pairing their outputs with gold corrections, so the method assumes those automatic errors look like the errors the LLM will see at test time.

Editorial extensions

If this is right

On all seven English-to-{De,Fr,Es,Ca,Ar,Tr} tasks, RefineBoth improves BLEU, COMET, and WER over the underlying ST/ASR system in every fine-tuned setting the paper reports.
Joint refinement outperforms translation-only refinement: for example, on MuST-C En→De with context-aware Mistral-12B, RefineBoth is 2.50 BLEU above RefineST.
Document-level context of K=3 sentences improves results over sentence-level refinement, and shuffling or swapping the context degrades performance, showing the model really uses discourse information.
Two-stage fine-tuning (generation followed by refinement) consistently beats single-stage refinement fine-tuning for both LLaMA3-8B and Mistral-12B.
Open 8B/12B LLMs fine-tuned with LoRA outperform closed GPT-3.5-turbo used for in-context learning on the reported metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the gains stem from mutual error correction, then using a better ASR or ST system should automatically produce better refinement training data, so the pipeline could be embedded in a self-improving loop.
A concrete next step the paper leaves implicit is to take the refined transcription A′ and retrain the ASR model on it, then measure whether second-round WER and BLEU improve further.
Because document context helps most in a targeted three-sentence window, discourse-sensitive errors such as pronoun choice and coreference are likely the next bottleneck; translation directions with more pronoun dropping may benefit most from wider context.
The latency cost of refinement suggests the practical payoff will be largest offline or in chunked live settings; testing RefineBoth on streaming partial chunks would be a natural extension.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The takeaway is that joint refinement—prompting an LLM to correct both the ASR transcript and the ST translation in one pass, with the transcript generated first—gives a reliable few-BLEU bump across seven language pairs, and the paper does enough ablation work that this is not a one-dataset fluke.

What's actually new is modest but real. Most prior LLM post-editing for speech translation refines the translation only; feeding both signals and generating the corrected transcript first lets the translation step condition on a cleaner source. The two-stage fine-tuning and the document-context variant are more than prompt gloss—they add up to a practical recipe. The evaluation deserves credit: three LLMs (GPT-3.5, LLaMA3-8B, Mistral-12B), both in-context learning and LoRA fine-tuning, a robustness check on ConST, an ASR-quality sweep, a context-shuffle test, and a GPT-4o judge. Code and data are promised on GitHub. That is a serious empirical package.

Now the soft spots, in order of real weight. First, the headline gains come without confidence intervals or significance tests against the baseline. The abstract says "significantly improved," but the only significance test reported is RefineBoth vs. RefineST, and the CoVoST 2 in-context-learning results are on a 500-sample subset. Given test set sizes (2,000+ sentences), a 3+ BLEU gain is probably real, but the paper should demonstrate it rather than assert it. Second, the fine-tuning data are generated by running the same CRESS model on its own MuST-C training split. Those outputs are in-sample and likely closer to gold than test-time outputs, so the LLM is trained to correct an easier error distribution. This is a genuine distribution-shift concern, and the ConST robustness experiment only partially answers it: it shows transfer across ST systems, but not that the training-split errors resemble test-split errors. Third, a small but telling citation error: the COMET metric is attributed to Bosselut et al. (the commonsense COMET); the right reference is Rei et al. That kind of mistake makes me want a careful pass over the rest of the bibliography.

Who is this for? Practitioners wiring an LLM post-editor into an ST pipeline, and researchers working on LLM-based correction of cascaded systems. It is not a fundamental advance, but it is a solid engineering contribution with reproducible results. I would send it to peer review, with the request that the authors add significance statements for the main comparisons and address the training-split distribution issue, either by analysis or by a caveat.

Referee Report

3 major / 6 minor

Summary. The paper proposes a post-hoc refinement pipeline in which an LLM receives both the ASR transcription and the ST translation produced by an automatic speech-translation system and jointly outputs a corrected transcription and translation (RefineBoth). This is contrasted with refining only the translation (RefineST) and with paraphrasing the translation without the transcription (ParaphraseST). The method is evaluated in three settings: in-context learning with GPT-3.5-turbo, context-agnostic parameter-efficient fine-tuning with LLaMA3-8B and Mistral-12B, and context-aware fine-tuning that concatenates K neighboring sentences. Experiments on MuST-C (En->De/Fr/Es) and CoVoST 2 (En->De/Ca/Ar/Tr) report BLEU, COMET, and WER gains over the CRESS, SpeechLM-P, and ConST baselines, with additional analyses covering semantic/fluency metrics, ASR quality sensitivity, context length, context shuffling, and GPT-4o judgments.

Significance. If the reported gains hold under more stringent evaluation, the paper makes a useful empirical contribution: it is, to my knowledge, one of the first systematic studies to apply LLM-based joint refinement to ASR and ST outputs, and it covers several LLMs, seven language pairs, and both in-context learning and fine-tuning. The authors release code and data, and the evaluation uses standard held-out test sets with external metrics (SacreBLEU, COMET, WER) as well as a GPT-4o judge, so there is no circularity burden in the core claim. The context-aware extension with chunk-based decoding and the robustness check on ConST are also valuable. The main limitations are statistical: the headline comparisons lack confidence intervals or significance tests, and the fine-tuning data are generated from in-sample outputs of the same ASR/ST systems used at test time.

major comments (3)

[Section IV-A, Tables II and III] The fine-tuning pairs are built by running the CRESS model (trained from scratch on MuST-C) and Whisper/SpeechLM-P on the training splits and pairing their outputs with gold references. Since CRESS is trained on exactly those MuST-C training sentences, its train-split ASR/ST outputs are in-sample and likely closer to the gold than its test-split outputs; the paper reports test WER 10.65 and BLEU 29.40 but gives no train-split error statistics. The LLM is therefore fine-tuned to correct an easier, possibly different error distribution, which is load-bearing for the PEFT half of the central claim. Please report train-split vs test-split WER/BLEU for the systems that generated the refinement pairs, and ideally add an experiment in which the refinement model is trained on pairs from a held-out or differently initialized ASR/ST system to demonstrate that the gains transfer to out-of-sample error patterns.
[Tables II and III, Section IV-B Q5] The abstract and Section IV-B state that the ST model is 'significantly improved,' but the main comparisons against the CRESS/SpeechLM-P baselines in Tables II and III are reported without confidence intervals or significance tests. The only significance test reported is the RefineBoth vs RefineST comparison (p<0.01 in Section IV-B Q5). Several headline gains are small in absolute terms (e.g., En->Fr zero-shot RefineBoth BLEU 40.22 vs 40.00 in Table II, and En->Es context-agnostic LLaMA3-8B COMET 0.8142 vs 0.7805), so without intervals it is unclear which of these differences are reliable. Please provide bootstrap confidence intervals or paired significance tests for the primary RefineBoth-vs-baseline results on both datasets.
[Section IV-C and Appendix A (Table XII)] For CoVoST 2, the in-context learning results are reported only on a randomly selected 500-sample subset of each test split (Appendix A), while the main Table III reports fine-tuning results. The abstract's claim of effectiveness 'across seven translation tasks' in the training-free scenario is therefore supported by a subsample for four of the seven tasks, with no confidence intervals on that subsample. Please state explicitly in the main text that the CoVoST 2 ICL numbers are on a 500-sample subset, and either run on the full test set or report intervals that quantify the sampling variability.

minor comments (6)

[Table II] The label 'CRESS (F&F)' is not defined in the table or the text; the surrounding text says the implementation matches Fang et al. [5], so please clarify whether 'F&F' refers to Fang and Feng [51] and define the label in the caption.
[References, [55]] Reference [55] cites Bosselut et al., 'COMET: Commonsense Transformers for automatic knowledge graph construction,' but the paper actually uses Unbabel/wmt22-comet-da for MT evaluation; the correct citation is Rei et al. (2020), 'COMET: A Neural Framework for MT Evaluation.' Please update the reference.
[Appendix B, Table XVI] In the ParaphraseST prompt example, the query sentence is 'You're going to see the whole thing take place in this movie,' which is English, but the task is to paraphrase the German ST output; this appears to be a typo that should be corrected for reproducibility.
[Section IV-A c] The text says 'We than concatenate [EA, ES]'; this should be 'then,' and the retrieval procedure would benefit from a one-sentence description of the L2 distance computation.
[Section V-F] The GPT-4o evaluation is based on 200 randomly selected samples across seven language pairs, so the per-pair sample size is about 29; please report the per-pair sample sizes and, if available, agreement statistics or significance of the score differences.
[Section IV-C] The sentence 'RefineBoth achieves the best result cross the four language pairs' should read 'across the four language pairs.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports empirical refinement gains measured on held-out test sets against external metrics; no claim reduces to its own inputs.

full rationale

The paper's central claim is empirical: jointly refining ASR transcription and ST translation with LLMs improves ST quality, evaluated with SacreBLEU, COMET, WER, BERTScore, perplexity, coherence, APT, and a GPT-4o judge on held-out MuST-C and CoVoST 2 test sets. The refinement training pairs are constructed by running ASR and ST models on training splits and pairing their outputs with gold references (Section IV-A); this is standard supervised data construction, not a fitted parameter renamed as a prediction. The RefineBoth versus RefineST comparison uses identical inputs and differs only by task definition, so the observed BLEU differences are measured outcomes rather than consequences of construction. No load-bearing step relies on a self-citation or on a uniqueness theorem; the CRESS and SpeechLM-P baselines are external systems, and the ConST experiment in Section V-G tests robustness to a different ST system. The only substantive weakness, that training-split ASR/ST errors may differ from test-time errors, is a generalization assumption about supervised learning, not a circularity in the derivation. Consequently, the paper is self-contained as an empirical evaluation and receives a score of 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities, particles, forces, or conserved quantities. It relies on existing LLMs and standard datasets. The only novel construction is the RefineBoth task/prompt, which is a data-format choice rather than an invented physical or mathematical entity.

free parameters (2)

Context length K = 3
Chosen by validation-set performance in Table VII; used for all context-aware fine-tuning.
LoRA rank = 8
Hyperparameter for parameter-efficient fine-tuning; chosen by the authors.

assumptions (4)

domain assumption LLM-based post-editing improves translation quality.
The method relies on prior results [7,8,40,42] showing LLM post-editing helps text-to-text translation.
ad hoc to paper Errors in ASR transcription and ST translation are complementary and can correct each other.
This is the paper's motivating hypothesis, illustrated with one example in Figure 1 and supported by case studies in Section V-H; it is not proven generally.
domain assumption Automatically generated refinement pairs from the training set generalize to the test set.
The fine-tuning data is created by running ASR/ST on training splits (Section IV-A); the model assumes the test-time error distribution is similar.
domain assumption Standard metrics (BLEU, COMET, WER) and GPT-4o judging reflect translation quality.
Evaluation uses SacreBLEU, COMET, WER, plus GPT-4o scores on 200 samples; metric validity is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Speech Translation Refinement using Large Language Models." pith.science (2026). https://pith.science/paper/ZGF2PU7B

@misc{pith2026250115090,
  author       = {Pith},
  title        = {Pith review of: Speech Translation Refinement using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZGF2PU7B}},
  note         = {Machine review of arXiv:2501.15090}
}

read the original abstract

Recent advancements in large language models (LLMs) have demonstrated their remarkable capabilities across various language tasks. Inspired by the success of text-to-text translation refinement, this paper investigates how LLMs can improve the performance of speech translation by introducing a joint refinement process. Through the joint refinement of speech translation (ST) and automatic speech recognition (ASR) transcription via LLMs, the performance of the ST model is significantly improved in both training-free in-context learning and parameter-efficient fine-tuning scenarios. Additionally, we explore the effect of document-level context on refinement under the context-aware fine-tuning scenario. Experimental results on the MuST-C and CoVoST 2 datasets, which include seven translation tasks, demonstrate the effectiveness of the proposed approach using several popular LLMs including GPT-3.5-turbo, LLaMA3-8B, and Mistral-12B. Further analysis further suggests that jointly refining both transcription and translation yields better performance compared to refining translation alone. Meanwhile, incorporating document-level context significantly enhances refinement performance. We release our code and datasets on GitHub.

Figures

Figures reproduced from arXiv: 2501.15090 by the authors.

**Figure 2.** Pipeline for the joint refinement. RefineBoth is on the right part of the pipeline (highlighted in gray). Query In-context examples (optional) Instruction [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Prompts for ST refinement using LLMs, including instruction, optional in-context examples (used only for in-context learning), and query. The [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Prompt used in the first stage of fine-tuning. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Prompt for GPT evaluation. follow Sun et al. [61] by deliberately introducing incorrect context. Specifically, we begin by shuffling the sentence order within each document and then reassembling it using CBD, which we term Local Shuffle, and then, by swapping sentences…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 59 canonical work pages

[1]

Lattice trans- former for speech translation,

P. Zhang, N. Ge, B. Chen, and K. Fan, “Lattice trans- former for speech translation,” in Proceedings of ACL , 2019, pp. 6475–6484

work page 2019
[2]

Speech translation and the end-to-end promise: Taking stock of where we are,

M. Sperber and M. Paulik, “Speech translation and the end-to-end promise: Taking stock of where we are,” in Proceedings of ACL , 2020, pp. 7409–7421

work page 2020
[3]

Cascaded mod- els with cyclic feedback for direct speech translation,

T. K. Lam, S. Schamoni, and S. Riezler, “Cascaded mod- els with cyclic feedback for direct speech translation,” in Proceedings of ICASSP , 2021, pp. 7508–7512

work page 2021
[4]

End-to-end speech trans- lation via cross-modal progressive training,

R. Ye, M. Wang, and L. Li, “End-to-end speech trans- lation via cross-modal progressive training,” in Proceed- ings of INTERSPEECH , 2021, pp. 2267–2271

work page 2021
[5]

STEMM: Self-learning with speech-text manifold mixup for speech translation,

Q. Fang, R. Ye, L. Li, Y . Feng, and M. Wang, “STEMM: Self-learning with speech-text manifold mixup for speech translation,” in Proceedings of ACL , 2022, pp. 7050– 7062

work page 2022
[6]

CKDST: Comprehensively and effectively distill knowledge from machine translation to end-to-end speech translation,

Y . Lei, Z. Xue, X. Zhao, H. Sun, S. Zhu, X. Lin, and D. Xiong, “CKDST: Comprehensively and effectively distill knowledge from machine translation to end-to-end speech translation,” in Findings of ACL, 2023, pp. 3123– 3137

work page 2023
[7]

Iterative translation refinement with large language models,

P. Chen, Z. Guo, B. Haddow, and K. Heafield, “Iterative translation refinement with large language models,” in Proceedings of EACL , 2024, pp. 181–190. [Online]. Available: https://aclanthology.org/2024.eamt-1.17

work page 2024
[8]

Leveraging gpt-4 for automatic translation post-editing,

V . Raunak, A. Sharaf, Y . Wang, H. H. Awadallah, and A. Menezes, “Leveraging gpt-4 for automatic translation post-editing,” in Findings of EMNLP , 2023, pp. 12 009– 12 024

work page 2023

Show all 63 references

[9]

LoRA: Low-rank adaptation of large language models,

E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in Proceedings of ICLR, 2022. 11 TABLE XIV A PROMPT EXAMPLE OF RE F I N EBoth , WITH GRAY AREAS INDICATING THE IN -CONTEXT LEARNING EXAM...

2022
[10]

Language models are few-shot learners,

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. ...

2020
[11]

The llama 3 herd of models,

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, and et al, “The llama 3 herd of models,” 2024

2024
[12]

Mistral nemo,

M. A. Team, “Mistral nemo,” 2024. [Online]. Available: https://mistral.ai/news/mistral-nemo/ 12

2024
[13]

Blending LLMs into cas- caded speech translation: KIT‘s offline speech translation system for IWSLT 2024,

S. Koneru, T. Binh Nguyen, N.-Q. Pham, D. Liu, Z. Li, A. Waibel, and J. Niehues, “Blending LLMs into cas- caded speech translation: KIT‘s offline speech translation system for IWSLT 2024,” in Proceedings of IWSLT) , Aug. 2024, pp. 183–191

2024
[14]

Cur- riculum pre-training for end-to-end speech translation,

C. Wang, Y . Wu, S. Liu, M. Zhou, and Z. Yang, “Cur- riculum pre-training for end-to-end speech translation,” in Proceedings of ACL, 2020, pp. 3728–3738

2020
[15]

Effectively pretraining a speech translation decoder with machine translation data,

A. Alinejad and A. Sarkar, “Effectively pretraining a speech translation decoder with machine translation data,” in Proceedings of EMNLP , 2020, pp. 8014–8020

2020
[16]

Unified speech-text pre-training for speech translation and recognition,

Y . Tang, H. Gong, N. Dong, C. Wang, W.-N. Hsu, J. Gu, A. Baevski, X. Li, A. Mohamed, M. Auli, and J. Pino, “Unified speech-text pre-training for speech translation and recognition,” inProceedings of ACL, 2022, pp. 1488– 1499

2022
[17]

SpeechUT: Bridging speech and text with hidden-unit for encoder-decoder based speech-text pre-training,

Z. Zhang, L. Zhou, J. Ao, S. Liu, L. Dai, J. Li, and F. Wei, “SpeechUT: Bridging speech and text with hidden-unit for encoder-decoder based speech-text pre-training,” in Proceedings of EMNLP , 2022, pp. 1663–1676

2022
[18]

Harnessing indirect training data for end- to-end automatic speech translation: Tricks of the trade,

J. Pino, L. Puzon, J. Gu, X. Ma, A. D. McCarthy, and D. Gopinath, “Harnessing indirect training data for end- to-end automatic speech translation: Tricks of the trade,” in Proceedings of IWSLT , 2019

2019
[19]

Self-Training for End-to-End Speech Translation,

J. Pino, Q. Xu, X. Ma, M. J. Dousti, and Y . Tang, “Self-Training for End-to-End Speech Translation,” in Proceedings of Interspeech , 2020, pp. 1476–1480

2020
[20]

Sample, translate, recombine: Leveraging audio alignments for data augmentation in end-to-end speech translation,

T. K. Lam, S. Schamoni, and S. Riezler, “Sample, translate, recombine: Leveraging audio alignments for data augmentation in end-to-end speech translation,” in Proceedings of ACL , 2022, pp. 245–254

2022
[21]

Cross-modal contrastive learning for speech translation,

R. Ye, M. Wang, and L. Li, “Cross-modal contrastive learning for speech translation,” in Proceedings of NAACL, 2022, pp. 5099–5113

2022
[22]

FCGCL: Fine- and coarse-granularity contrastive learning for speech translation,

H. Zhang, N. Si, Y . Chen, Z. Li, T. Niu, X. Yang, and D. Qu, “FCGCL: Fine- and coarse-granularity contrastive learning for speech translation,” in Findings of EMNLP , 2022, pp. 3048–3059

2022
[23]

W ACO: Word-aligned con- trastive learning for speech translation,

S. Ouyang, R. Ye, and L. Li, “W ACO: Word-aligned con- trastive learning for speech translation,” in Proceedings of ACL, 2023, pp. 3891–3907

2023
[24]

Improving speech translation by fusing speech and text,

W. Yin, Z. Liu, C. Zhao, T. Wang, J. Tong, and R. Ye, “Improving speech translation by fusing speech and text,” in Findings of EMNLP , 2023, pp. 6262–6273

2023
[25]

A simple con- catenation can effectively improve speech translation,

L. Zhang, K. Fan, B. Chen, and L. Si, “A simple con- catenation can effectively improve speech translation,” in Proceedings of ACL , 2023, pp. 1793–1802

2023
[26]

CMOT: Cross-modal mixup via optimal transport for speech translation,

Y . Zhou, Q. Fang, and Y . Feng, “CMOT: Cross-modal mixup via optimal transport for speech translation,” in Proceedings of ACL , 2023, pp. 7873–7887

2023
[27]

Im- proving speech translation by understanding and learning from the auxiliary text translation task,

Y . Tang, J. Pino, X. Li, C. Wang, and D. Genzel, “Im- proving speech translation by understanding and learning from the auxiliary text translation task,” in Proceedings of ACL, 2021, pp. 4252–4261

2021
[28]

Modality adaption or regularization? a case study on end-to-end speech translation,

Y . Han, C. Xu, T. Xiao, and J. Zhu, “Modality adaption or regularization? a case study on end-to-end speech translation,” in Proceedings of ACL , 2023, pp. 1340– 1348

2023
[29]

An empirical study of consistency regularization for end-to- end speech-to-text translation,

P. Gao, R. Zhang, Z. He, H. Wu, and H. Wang, “An empirical study of consistency regularization for end-to- end speech-to-text translation,” in Proceedings of ACL , 2024, pp. 242–256

2024
[30]

On decoder-only architecture for speech-to-text and large language model integration,

J. Wu, Y . Gaur, Z. Chen, L. Zhou, Y . Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu, and Y . Wu, “On decoder-only architecture for speech-to-text and large language model integration,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , 2023, pp. 1–8

2023
[31]

Salm: Speech-augmented language model with in- context learning for speech recognition and translation,

Z. Chen, H. Huang, A. Andrusenko, O. Hrinchuk, K. C. Puvvada, J. Li, S. Ghosh, J. Balam, and B. Gins- burg, “Salm: Speech-augmented language model with in- context learning for speech recognition and translation,” in Proceedings of ICASSP , 2024, pp. 13 521–13 525

2024
[32]

SALMONN: towards generic hearing abilities for large language models,

C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “SALMONN: towards generic hearing abilities for large language models,” in Proceedings of ICLR , 2024

2024
[33]

Qwen-audio: Advancing univer- sal audio understanding via unified large-scale audio- language models,

Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing univer- sal audio understanding via unified large-scale audio- language models,” arXiv preprint arXiv:2311.07919 , 2023

2023 arXiv
[34]

Qwen2- audio technical report,

Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Lin, C. Zhou, and J. Zhou, “Qwen2- audio technical report,” arXiv preprint arXiv:2407.10759, 2024

2024 arXiv
[35]

AudioChatLlama: Towards general-purpose speech abilities for LLMs,

Y . Fathullah, C. Wu, E. Lakomkin, K. Li, J. Jia, Y . Shangguan, J. Mahadeokar, O. Kalinli, C. Fuegen, and M. Seltzer, “AudioChatLlama: Towards general-purpose speech abilities for LLMs,” in Proceedings of NAACL , Jun. 2024, pp. 5522–5532

2024
[36]

Speech translation with speech foundation models and large language models: What is there and what is missing?

M. Gaido, S. Papi, M. Negri, and L. Bentivogli, “Speech translation with speech foundation models and large language models: What is there and what is missing?” in Proceedings of ACL, Aug. 2024, pp. 14 760–14 778

2024
[37]

GenTranslate: Large language models are generative multilingual speech and machine translators,

Y . Hu, C. Chen, C.-H. Yang, R. Li, D. Zhang, Z. Chen, and E. Chng, “GenTranslate: Large language models are generative multilingual speech and machine translators,” in Proceedings of ACL, Aug. 2024, pp. 74–90

2024
[38]

Findings of the WMT 2018 shared task on automatic post-editing,

R. Chatterjee, M. Negri, R. Rubino, and M. Turchi, “Findings of the WMT 2018 shared task on automatic post-editing,” in Proceedings of WMT , 2018, pp. 710– 725

2018
[39]

Lexical translation inconsistency-aware document-level translation repair,

Z. Zhang, J. Li, S. Tao, and H. Yang, “Lexical translation inconsistency-aware document-level translation repair,” in Findings of ACL , 2023, pp. 12 492–12 505

2023
[40]

Improving llm-based machine translation with systematic self-correction,

Z. Feng, Y . Zhang, H. Li, W. Liu, J. Lang, Y . Feng, J. Wu, and Z. Liu, “Improving llm-based machine translation with systematic self-correction,” Computing Research Repository, vol. arXiv:2402.16379, 2024

2024 arXiv
[41]

Guiding large language models to post-edit machine translation with error annotations,

D. Ki and M. Carpuat, “Guiding large language models to post-edit machine translation with error annotations,” in Findings of NAACL, 2024

2024
[42]

Contex- tual refinement of translations: Large language models for sentence and document-level post-editing,

S. Koneru, M. Exel, M. Huck, and J. Niehues, “Contex- tual refinement of translations: Large language models for sentence and document-level post-editing,” in Pro- ceedings of NAACL , 2024, pp. 2711–2725. 13

2024
[43]

Hubert: Self- supervised speech representation learning by masked prediction of hidden units,

W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self- supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Trans. Audio, Speech and Lang. Proc. , vol. 29, p. 3451–3460, 2021

2021
[44]

Robust speech recog- nition via large-scale weak supervision,

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, “Robust speech recog- nition via large-scale weak supervision,” in Proceedings of ICML, 2023, pp. 28 492–28 518

2023
[45]

Don’t stop pretraining: Adapt language models to domains and tasks,

S. Gururangan, A. Marasovi ´c, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith, “Don’t stop pretraining: Adapt language models to domains and tasks,” in Proceedings of ACL , 2020, pp. 8342–8360

2020
[46]

Neural machine transla- tion with extended context,

J. Tiedemann and Y . Scherrer, “Neural machine transla- tion with extended context,” in Proceedings of the Third Workshop on Discourse in Machine Translation , Sep. 2017, pp. 82–92

2017
[47]

P-Transformer: Towards Better Document-to-Document Neural Machine Translation,

Y . Li, J. Li, J. Jiang, S. Tao, H. Yang, and M. Zhang, “P-Transformer: Towards Better Document-to-Document Neural Machine Translation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 31, pp. 3859–3870, 2023

2023
[48]

Beyond sentence-level end-to-end speech translation: Context helps,

B. Zhang, I. Titov, B. Haddow, and R. Sennrich, “Beyond sentence-level end-to-end speech translation: Context helps,” in Proceedings of ACL , 2021, pp. 2566–2578

2021
[49]

MuST-C: a Multilingual Speech Translation Corpus,

M. A. Di Gangi, R. Cattoni, L. Bentivogli, M. Negri, and M. Turchi, “MuST-C: a Multilingual Speech Translation Corpus,” in Proceedings of NAACL , 2019, pp. 2012– 2017

2019
[50]

Covost 2: A massively multilingual speech-to-text translation corpus,

C. Wang, A. Wu, and J. Pino, “Covost 2: A massively multilingual speech-to-text translation corpus,” Comput- ing Research Repository , vol. arXiv:2007.10310, 2020

2007 arXiv
[51]

Understanding and bridging the modality gap for speech translation,

Q. Fang and Y . Feng, “Understanding and bridging the modality gap for speech translation,” in Proceedings of ACL, 2023, pp. 15 864–15 881

2023
[52]

Speechlm: Enhanced speech pre-training with unpaired textual data,

Z. Zhang, S. Chen, L. Zhou, Y . Wu, S. Ren, S. Liu, Z. Yao, X. Gong, L. Dai, J. Li, and F. Wei, “Speechlm: Enhanced speech pre-training with unpaired textual data,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2177–2187, 2024

2024
[53]

LlamaFactory: Unified efficient fine-tuning of 100+ lan- guage models,

Y . Zheng, R. Zhang, J. Zhang, Y . Ye, and Z. Luo, “LlamaFactory: Unified efficient fine-tuning of 100+ lan- guage models,” in Proceedings of ACL: System Demon- strations, 2024, pp. 400–410

2024
[54]

A call for clarity in reporting BLEU scores,

M. Post, “A call for clarity in reporting BLEU scores,” in Proceedings of WMT , 2018, pp. 186–191

2018
[55]

COMET: Commonsense trans- formers for automatic knowledge graph construction,

A. Bosselut, H. Rashkin, M. Sap, C. Malaviya, A. Ce- likyilmaz, and Y . Choi, “COMET: Commonsense trans- formers for automatic knowledge graph construction,” in Proceedings of ACL , 2019, pp. 4762–4779

2019
[56]

Statistical significance tests for machine trans- lation evaluation,

P. Koehn, “Statistical significance tests for machine trans- lation evaluation,” in Proceedings of EMNLP , 2004, pp. 388–395

2004
[57]

Bertscore: Evaluating text generation with bert,

T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” in proceedings of ICLR , 2020

2020
[58]

Con- trastive decoding: Open-ended text generation as opti- mization,

X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. Hashimoto, L. Zettlemoyer, and M. Lewis, “Con- trastive decoding: Open-ended text generation as opti- mization,” in Proceedings of ACL , 2023, pp. 12 286– 12 312

2023
[59]

Language models are unsupervised multitask learners,

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” OpenAI blog, 2019

2019
[60]

SimCSE: Simple con- trastive learning of sentence embeddings,

T. Gao, X. Yao, and D. Chen, “SimCSE: Simple con- trastive learning of sentence embeddings,” inProceedings of EMNLP, Nov. 2021, pp. 6894–6910

2021
[61]

Rethinking document-level neural machine translation,

Z. Sun, M. Wang, H. Zhou, C. Zhao, S. Huang, J. Chen, and L. Li, “Rethinking document-level neural machine translation,” in Findings of ACL , 2022, pp. 3537–3548

2022
[62]

Validation of an automatic metric for the accuracy of pronoun translation (APT),

L. Miculicich Werlen and A. Popescu-Belis, “Validation of an automatic metric for the accuracy of pronoun translation (APT),” in Proceedings of DiscoMT , 2017, pp. 17–25

2017
[63]

Large language models are state-of-the-art evaluators of translation quality,

T. Kocmi and C. Federmann, “Large language models are state-of-the-art evaluators of translation quality,” in Proceedings of EAMT , 2023, pp. 193–203

2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Lattice trans- former for speech translation,

P. Zhang, N. Ge, B. Chen, and K. Fan, “Lattice trans- former for speech translation,” in Proceedings of ACL , 2019, pp. 6475–6484

work page 2019

[2] [2]

Speech translation and the end-to-end promise: Taking stock of where we are,

M. Sperber and M. Paulik, “Speech translation and the end-to-end promise: Taking stock of where we are,” in Proceedings of ACL , 2020, pp. 7409–7421

work page 2020

[3] [3]

Cascaded mod- els with cyclic feedback for direct speech translation,

T. K. Lam, S. Schamoni, and S. Riezler, “Cascaded mod- els with cyclic feedback for direct speech translation,” in Proceedings of ICASSP , 2021, pp. 7508–7512

work page 2021

[4] [4]

End-to-end speech trans- lation via cross-modal progressive training,

R. Ye, M. Wang, and L. Li, “End-to-end speech trans- lation via cross-modal progressive training,” in Proceed- ings of INTERSPEECH , 2021, pp. 2267–2271

work page 2021

[5] [5]

STEMM: Self-learning with speech-text manifold mixup for speech translation,

Q. Fang, R. Ye, L. Li, Y . Feng, and M. Wang, “STEMM: Self-learning with speech-text manifold mixup for speech translation,” in Proceedings of ACL , 2022, pp. 7050– 7062

work page 2022

[6] [6]

CKDST: Comprehensively and effectively distill knowledge from machine translation to end-to-end speech translation,

Y . Lei, Z. Xue, X. Zhao, H. Sun, S. Zhu, X. Lin, and D. Xiong, “CKDST: Comprehensively and effectively distill knowledge from machine translation to end-to-end speech translation,” in Findings of ACL, 2023, pp. 3123– 3137

work page 2023

[7] [7]

Iterative translation refinement with large language models,

P. Chen, Z. Guo, B. Haddow, and K. Heafield, “Iterative translation refinement with large language models,” in Proceedings of EACL , 2024, pp. 181–190. [Online]. Available: https://aclanthology.org/2024.eamt-1.17

work page 2024

[8] [8]

Leveraging gpt-4 for automatic translation post-editing,

V . Raunak, A. Sharaf, Y . Wang, H. H. Awadallah, and A. Menezes, “Leveraging gpt-4 for automatic translation post-editing,” in Findings of EMNLP , 2023, pp. 12 009– 12 024

work page 2023