{"id":"11acc408-9a91-454c-bb97-886df31ef315","arxiv_id":"2501.15090","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Jointly refining ASR transcripts and ST translations with an LLM improves BLEU and COMET on MuST-C and CoVoST 2 across seven translation tasks.","lead":"The paper shows that asking a large language model to fix both the speech-recognition transcript and the machine translation together improves speech translation quality. This offers a practical way to boost speech translation systems without retraining the core model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning refinement data are generated by running CRESS on its own MuST-C training split, so the LLM learns to correct in-sample errors that are likely easier than test-time errors; the reported PEFT gains may not transfer.","rationale":"The reader's weakest assumption identifies exactly the point on which the fine-tuning results depend: the refinement LLM is trained on ASR/ST outputs produced by the same base models on the same training data those models were trained on. Because the paper reports no training-split quality numbers, there is no way to tell whether the errors the LLM learns to correct resemble test errors. This is a generalization risk rather than an internal inconsistency: the numbers in Tables II and III are what they are, and the ConST experiments provide partial evidence of robustness. But the central claim is specifically that joint refinement 'significantly improves' ST performance, and if the synthetic training distribution is not representative, the magnitude of the improvement on any held-out or real-world input is uncertain. The proposed check, retraining CRESS on half the data and generating refinement pairs on the other half, would isolate the in-sample error confound. Since this concern is the same as the reader's, and since it warrants the paper providing additional evidence rather than being definitive enough to reject, the appropriate verdict remains CONDITIONAL. No significant additional objection was found beyond this one; the paper's multi-scenario evaluation, use of multiple LLMs, and ConST robustness test are genuine strengths that keep the claim plausible pending this check.","tokens_in":19805,"tokens_out":12408,"duration_ms":126725,"concrete_test":"Retrain the CRESS baseline on only half of the MuST-C training set; use that checkpoint to generate ASR/ST outputs on the other held-out half; fine-tune Mistral-12B RefineBoth on those held-out pairs; then evaluate on the same tst-COMMON test set. If the BLEU/COMET gains over CRESS shrink or disappear compared with the paper's full-train refinement setup, the reported gains are partly an artifact of training on in-sample CRESS errors. A cheaper preliminary check: report CRESS's WER and BLEU on a random 5k-sentence subset of its training split; a large gap versus the test split would confirm the distribution mismatch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the PEFT half of the central claim is that the synthetic refinement pairs built in Section IV-A are representative of test-time ASR/ST errors. The pairs are produced by running the same CRESS model (trained from scratch on MuST-C, Section IV-A b) and Whisper/SpeechLM-P on the *training* splits, then pairing outputs with gold references. Because CRESS was trained on exactly those MuST-C training sentences, its train-split ASR/ST outputs are in-sample and likely closer to gold than the test-split outputs (the paper reports test WER 10.65 and BLEU 29.40 but no train-split numbers). The LLM is therefore fine-tuned to correct an easier, possibly different error distribution. The observed test improvements (e.g., Mistral-12B RefineBoth 32.75 BLEU vs 29.40 baseline in Table II) could be either conservative or inflated relative to deployment, but the paper provides no evidence about which. The ConST experiments in Section V-G show robustness to a different ST model, but they do not address the in-sample-vs-out-of-sample error mismatch, because those refinement pairs are also generated on the training split. This is the weakest point in the causal chain from 'joint refinement helps on these benchmarks' to 'joint refinement improves ST models in general.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post-hoc refinement pipeline in which an LLM receives both the ASR transcription and the ST translation produced by an automatic speech-translation system and jointly outputs a corrected transcription and translation (RefineBoth). This is contrasted with refining only the translation (RefineST) and with paraphrasing the translation without the transcription (ParaphraseST). The method is evaluated in three settings: in-context learning with GPT-3.5-turbo, context-agnostic parameter-efficient fine-tuning with LLaMA3-8B and Mistral-12B, and context-aware fine-tuning that concatenates K neighboring sentences. Experiments on MuST-C (En->De/Fr/Es) and CoVoST 2 (En->De/Ca/Ar/Tr) report BLEU, COMET, and WER gains over the CRESS, SpeechLM-P, and ConST baselines, with additional analyses covering semantic/fluency metrics, ASR quality sensitivity, context length, context shuffling, and GPT-4o judgments.","tokens_in":20054,"tokens_out":4755,"duration_ms":46506,"significance":"If the reported gains hold under more stringent evaluation, the paper makes a useful empirical contribution: it is, to my knowledge, one of the first systematic studies to apply LLM-based joint refinement to ASR and ST outputs, and it covers several LLMs, seven language pairs, and both in-context learning and fine-tuning. The authors release code and data, and the evaluation uses standard held-out test sets with external metrics (SacreBLEU, COMET, WER) as well as a GPT-4o judge, so there is no circularity burden in the core claim. The context-aware extension with chunk-based decoding and the robustness check on ConST are also valuable. The main limitations are statistical: the headline comparisons lack confidence intervals or significance tests, and the fine-tuning data are generated from in-sample outputs of the same ASR/ST systems used at test time.","major_comments":[{"comment":"The fine-tuning pairs are built by running the CRESS model (trained from scratch on MuST-C) and Whisper/SpeechLM-P on the training splits and pairing their outputs with gold references. Since CRESS is trained on exactly those MuST-C training sentences, its train-split ASR/ST outputs are in-sample and likely closer to the gold than its test-split outputs; the paper reports test WER 10.65 and BLEU 29.40 but gives no train-split error statistics. The LLM is therefore fine-tuned to correct an easier, possibly different error distribution, which is load-bearing for the PEFT half of the central claim. Please report train-split vs test-split WER/BLEU for the systems that generated the refinement pairs, and ideally add an experiment in which the refinement model is trained on pairs from a held-out or differently initialized ASR/ST system to demonstrate that the gains transfer to out-of-sample error patterns.","section":"Section IV-A, Tables II and III"},{"comment":"The abstract and Section IV-B state that the ST model is 'significantly improved,' but the main comparisons against the CRESS/SpeechLM-P baselines in Tables II and III are reported without confidence intervals or significance tests. The only significance test reported is the RefineBoth vs RefineST comparison (p<0.01 in Section IV-B Q5). Several headline gains are small in absolute terms (e.g., En->Fr zero-shot RefineBoth BLEU 40.22 vs 40.00 in Table II, and En->Es context-agnostic LLaMA3-8B COMET 0.8142 vs 0.7805), so without intervals it is unclear which of these differences are reliable. Please provide bootstrap confidence intervals or paired significance tests for the primary RefineBoth-vs-baseline results on both datasets.","section":"Tables II and III, Section IV-B Q5"},{"comment":"For CoVoST 2, the in-context learning results are reported only on a randomly selected 500-sample subset of each test split (Appendix A), while the main Table III reports fine-tuning results. The abstract's claim of effectiveness 'across seven translation tasks' in the training-free scenario is therefore supported by a subsample for four of the seven tasks, with no confidence intervals on that subsample. Please state explicitly in the main text that the CoVoST 2 ICL numbers are on a 500-sample subset, and either run on the full test set or report intervals that quantify the sampling variability.","section":"Section IV-C and Appendix A (Table XII)"}],"minor_comments":[{"comment":"The label 'CRESS (F&F)' is not defined in the table or the text; the surrounding text says the implementation matches Fang et al. [5], so please clarify whether 'F&F' refers to Fang and Feng [51] and define the label in the caption.","section":"Table II"},{"comment":"Reference [55] cites Bosselut et al., 'COMET: Commonsense Transformers for automatic knowledge graph construction,' but the paper actually uses Unbabel/wmt22-comet-da for MT evaluation; the correct citation is Rei et al. (2020), 'COMET: A Neural Framework for MT Evaluation.' Please update the reference.","section":"References, [55]"},{"comment":"In the ParaphraseST prompt example, the query sentence is 'You're going to see the whole thing take place in this movie,' which is English, but the task is to paraphrase the German ST output; this appears to be a typo that should be corrected for reproducibility.","section":"Appendix B, Table XVI"},{"comment":"The text says 'We than concatenate [EA, ES]'; this should be 'then,' and the retrieval procedure would benefit from a one-sentence description of the L2 distance computation.","section":"Section IV-A c"},{"comment":"The GPT-4o evaluation is based on 200 randomly selected samples across seven language pairs, so the per-pair sample size is about 29; please report the per-pair sample sizes and, if available, agreement statistics or significance of the score differences.","section":"Section V-F"},{"comment":"The sentence 'RefineBoth achieves the best result cross the four language pairs' should read 'across the four language pairs.'","section":"Section IV-C"}],"recommendation":"major_revision","confidential_remarks":"The in-sample training-data concern is the main correctness risk: the PEFT results could be optimistic because the refinement model learns to correct errors of a system on its own training split. This is not a circularity problem in the evaluation of the benchmarks, but it is a generalization gap that should be quantified or addressed before the paper's broader claims are accepted. The paper's novelty is modest but real, and the release of code and data is a strength. If the authors add train-vs-test error statistics, a cross-system robustness experiment, and confidence intervals for the primary comparisons, the paper would be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The takeaway is that joint refinement—prompting an LLM to correct both the ASR transcript and the ST translation in one pass, with the transcript generated first—gives a reliable few-BLEU bump across seven language pairs, and the paper does enough ablation work that this is not a one-dataset fluke.\n\nWhat's actually new is modest but real. Most prior LLM post-editing for speech translation refines the translation only; feeding both signals and generating the corrected transcript first lets the translation step condition on a cleaner source. The two-stage fine-tuning and the document-context variant are more than prompt gloss—they add up to a practical recipe. The evaluation deserves credit: three LLMs (GPT-3.5, LLaMA3-8B, Mistral-12B), both in-context learning and LoRA fine-tuning, a robustness check on ConST, an ASR-quality sweep, a context-shuffle test, and a GPT-4o judge. Code and data are promised on GitHub. That is a serious empirical package.\n\nNow the soft spots, in order of real weight. First, the headline gains come without confidence intervals or significance tests against the baseline. The abstract says \"significantly improved,\" but the only significance test reported is RefineBoth vs. RefineST, and the CoVoST 2 in-context-learning results are on a 500-sample subset. Given test set sizes (2,000+ sentences), a 3+ BLEU gain is probably real, but the paper should demonstrate it rather than assert it. Second, the fine-tuning data are generated by running the same CRESS model on its own MuST-C training split. Those outputs are in-sample and likely closer to gold than test-time outputs, so the LLM is trained to correct an easier error distribution. This is a genuine distribution-shift concern, and the ConST robustness experiment only partially answers it: it shows transfer across ST systems, but not that the training-split errors resemble test-split errors. Third, a small but telling citation error: the COMET metric is attributed to Bosselut et al. (the commonsense COMET); the right reference is Rei et al. That kind of mistake makes me want a careful pass over the rest of the bibliography.\n\nWho is this for? Practitioners wiring an LLM post-editor into an ST pipeline, and researchers working on LLM-based correction of cascaded systems. It is not a fundamental advance, but it is a solid engineering contribution with reproducible results. I would send it to peer review, with the request that the authors add significance statements for the main comparisons and address the training-split distribution issue, either by analysis or by a caveat.","headline":"Joint ASR+ST refinement is a real, reproducible win for LLM-based speech translation post-editing, but missing confidence intervals and in-sample training data keep it from being a clean bill of health.","tokens_in":20609,"tokens_out":4331,"would_cite":true,"duration_ms":38756,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Refining speech-to-text translation and ASR transcription together, in one LLM pass, consistently improves translation quality beyond fixing the translation alone.","keywords":["speech translation","LLM refinement","joint refinement","automatic speech recognition","in-context learning","parameter-efficient fine-tuning","document-level context","post-editing"],"falsifier":"Run the fine-tuned RefineBoth model on test audio with a new domain shift (for example, telephonic noise or an unseen accent) that creates ASR error types absent from the training split while keeping the ST system fixed; if BLEU drops below the translation-only RefineST baseline, the joint-refinement claim collapses. Alternatively, if shuffling sentences within a document produces no BLEU drop, the claimed context benefit is not real.","tokens_in":19590,"feed_emoji":"🎙️","tokens_out":5766,"duration_ms":53540,"temperature":0.7,"pith_summary":"The paper proposes that an LLM should fix both the automatic speech recognition output and the speech translation output at the same time, rather than polishing the translation by itself. On MuST-C and CoVoST 2 covering seven English-to-X tasks, this joint refinement raises translation-quality scores by up to 4.81 BLEU and 0.0755 COMET over the unrefined system and improves transcription WER. The gains appear in both in-context learning with GPT-3.5-turbo and in parameter-efficient fine-tuning of LLaMA3-8B and Mistral-12B, and document-level context adds a further boost. The reason the approach matters is that speech translation errors often stem from source transcription errors, so correcting both lets each output repair the other.","feed_headline":"Joint LLM refinement lifts speech translation by up to 4.8 BLEU","feed_subtitle":"Correcting both the transcript and the translation at once outperforms fixing the translation alone on seven language pairs.","key_machinery":"The load-bearing object is the RefineBoth task and its prompt format: an instruction, optional in-context examples, and a query that asks the LLM to output a refined transcription followed by a refined translation. Fine-tuning uses a two-stage procedure in which the LLM first learns to generate the ASR transcription and ST translation, then learns to refine both, and document-level context is handled by chunk-based decoding that concatenates K neighboring sentences with sentence indices to preserve alignment.","core_discovery":"The central discovery is that errors in ASR transcription and ST translation are complementary, not independent, and an LLM can exploit this. The paper defines the joint refinement task (A,S)→(A′,S′), where A is the noisy transcription and S the noisy translation, and the LLM first generates a corrected transcription A′ and then a corrected translation S′. Controlled comparisons show that this RefineBoth task beats RefineST, which only refines the translation, on BLEU across all evaluated language pairs, with differences reported as statistically significant at p<0.01, while COMET is similar or better. The paper further claims that a two-stage fine-tuning strategy — first teaching the LLM to produce A and S, then to refine both — outperforms refinement-only fine-tuning, and that incorporating document-level context of three sentences gives additional gains.","pith_inferences":["If the gains stem from mutual error correction, then using a better ASR or ST system should automatically produce better refinement training data, so the pipeline could be embedded in a self-improving loop.","A concrete next step the paper leaves implicit is to take the refined transcription A′ and retrain the ASR model on it, then measure whether second-round WER and BLEU improve further.","Because document context helps most in a targeted three-sentence window, discourse-sensitive errors such as pronoun choice and coreference are likely the next bottleneck; translation directions with more pronoun dropping may benefit most from wider context.","The latency cost of refinement suggests the practical payoff will be largest offline or in chunked live settings; testing RefineBoth on streaming partial chunks would be a natural extension."],"forward_implications":["On all seven English-to-{De,Fr,Es,Ca,Ar,Tr} tasks, RefineBoth improves BLEU, COMET, and WER over the underlying ST/ASR system in every fine-tuned setting the paper reports.","Joint refinement outperforms translation-only refinement: for example, on MuST-C En→De with context-aware Mistral-12B, RefineBoth is 2.50 BLEU above RefineST.","Document-level context of K=3 sentences improves results over sentence-level refinement, and shuffling or swapping the context degrades performance, showing the model really uses discourse information.","Two-stage fine-tuning (generation followed by refinement) consistently beats single-stage refinement fine-tuning for both LLaMA3-8B and Mistral-12B.","Open 8B/12B LLMs fine-tuned with LoRA outperform closed GPT-3.5-turbo used for in-context learning on the reported metrics."],"supporting_citations":[{"why":"CRESS is the end-to-end ST model used to generate the automatic translations and transcriptions on MuST-C.","marker":"[51]"},{"why":"SpeechLM-P generates the automatic speech translations on CoVoST 2.","marker":"[52]"},{"why":"Whisper large-v3 produces the ASR transcriptions for CoVoST 2.","marker":"[44]"},{"why":"Provides the text-to-text iterative translation refinement method that motivates this paper's approach.","marker":"[7]"},{"why":"Demonstrates LLM-based post-editing of machine translation, the prior result being extended to speech translation.","marker":"[8]"},{"why":"Shows that fine-tuning LLMs for post-editing improves sentence- and document-level translation, the baseline for the fine-tuning scenarios.","marker":"[42]"},{"why":"Provides chunk-based decoding, which the paper adapts for document-level context-aware refinement.","marker":"[48]"},{"why":"ConST is the alternative ST system used to show that joint refinement generalizes across different translation qualities.","marker":"[21]"},{"why":"LoRA is the parameter-efficient fine-tuning method applied to LLaMA3-8B and Mistral-12B.","marker":"[9]"}],"fun_headline_variants":["Fixing both transcript and translation boosts ST more than translation alone","LLM joint refinement of ASR and ST beats translation-only refinement","Same-pass transcript+translation refinement lifts speech translation BLEU","Correcting ASR text and translation together improves speech translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The refinement training data is made by running the same ASR and ST systems on the training set and pairing their outputs with gold corrections, so the method assumes those automatic errors look like the errors the LLM will see at test time.","fun_headline_variants_meta":{"raw":{"variants":["Fixing both transcript and translation boosts ST more than translation alone","LLM joint refinement of ASR and ST beats translation-only refinement","Same-pass transcript+translation refinement lifts speech translation BLEU","Correcting ASR text and translation together improves speech translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000464,"raw_usage":{"total_tokens":2296,"prompt_tokens":901,"completion_tokens":1395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1323}},"tokens_in":517,"tokens_out":1395,"duration_ms":9185,"temperature":1.0,"reasoning_tokens":1323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:37:29.399596+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the fine-tuned RefineBoth model on test audio with a new domain shift (for example, telephonic noise or an unseen accent) that creates ASR error types absent from the training split while keeping the ST system fixed; if BLEU drops below the translation-only RefineST baseline, the joint-refinement claim collapses. Alternatively, if shuffling sentences within a document produces no BLEU drop, the claimed context benefit is not real.","supporting_citations":[{"cited_title":"Understanding and bridging the modality gap for speech translation,","cited_arxiv_id":null,"evidence_quote":"CRESS is the end-to-end ST model used to generate the automatic translations and transcriptions on MuST-C."},{"cited_title":"Speechlm: Enhanced speech pre-training with unpaired textual data,","cited_arxiv_id":null,"evidence_quote":"SpeechLM-P generates the automatic speech translations on CoVoST 2."},{"cited_title":"Robust speech recog- nition via large-scale weak supervision,","cited_arxiv_id":null,"evidence_quote":"Whisper large-v3 produces the ASR transcriptions for CoVoST 2."},{"cited_title":"Iterative translation refinement with large language models,","cited_arxiv_id":null,"evidence_quote":"Provides the text-to-text iterative translation refinement method that motivates this paper's approach."},{"cited_title":"Leveraging gpt-4 for automatic translation post-editing,","cited_arxiv_id":null,"evidence_quote":"Demonstrates LLM-based post-editing of machine translation, the prior result being extended to speech translation."},{"cited_title":"Contex- tual refinement of translations: Large language models for sentence and document-level post-editing,","cited_arxiv_id":null,"evidence_quote":"Shows that fine-tuning LLMs for post-editing improves sentence- and document-level translation, the baseline for the fine-tuning scenarios."},{"cited_title":"Beyond sentence-level end-to-end speech translation: Context helps,","cited_arxiv_id":null,"evidence_quote":"Provides chunk-based decoding, which the paper adapts for document-level context-aware refinement."},{"cited_title":"Cross-modal contrastive learning for speech translation,","cited_arxiv_id":null,"evidence_quote":"ConST is the alternative ST system used to show that joint refinement generalizes across different translation qualities."},{"cited_title":"LoRA: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"LoRA is the parameter-efficient fine-tuning method applied to LLaMA3-8B and Mistral-12B."}],"review_version":1}