{"id":"b50d5a1d-9f4c-4f72-a915-e280d29d3ea0","arxiv_id":"2506.10299","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Slowly decaying the fraction of aligned text interleaved with speech units during LLM fine-tuning improves speech-to-speech translation, most for low-resource languages.","lead":"A fine-tuning schedule that mixes aligned text into speech-token training sequences and then gradually removes it improves speech-to-speech translation scores in all seven language pairs tested, with the largest gains for low-resource pairs. It offers a cheap, architecture-free training trick that helps text-only LLMs learn to speak, a core bottleneck for spoken translation and other speech-AI applications.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The only non-scheduled control (constant p=0.3) confounds scheduling with average text ratio, so the paper's central claim that the decay schedule helps is not empirically supported; a matched constant-p control is needed.","rationale":"I read the paper's central claim as comprising two statements: (1) interleaved speech-text training improves S2ST, and (2) the scheduled decay is the reason. Statement (1) is supported by Table 2 versus baseline, though without significance testing. Statement (2) is the method's novelty and is supported only by the p=0.3 constant control, which is confounded with average text ratio. This is more load-bearing than the alignment concern because even the 'without alignment' ablation in Table 3 still improves over baseline (11.5 vs 10.3), whereas if a matched constant-p control also works, the proposed method reduces to SpiRitLM-style interleaving and the 'scheduled' contribution disappears. The alignment concern is real but secondary; it would affect the size of the gains, not the existence of the core scheduling effect. The paper also lacks multiple seeds and confidence intervals, but the schedule confound is a specific, addressable flaw. I recommend keeping CONDITIONAL: the revision must add a matched constant-p (and ideally a shuffled-schedule) control before the scheduling claim can be accepted.","tokens_in":9622,"tokens_out":9551,"duration_ms":117319,"concrete_test":"Run a constant-p control on Pt-en with p set to the scheduled run's time-averaged effective text ratio (measured by instrumenting Algorithm 1 to log the actual fraction of replaced tokens at each step, including the overshoot effect of the stopping rule), keeping all other hyperparameters identical; if this control matches or beats the scheduled run's 19.5 ASR-BLEU, the scheduling itself is not the active ingredient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2's comparison that is supposed to isolate 'scheduling' compares Scheduled ILT (p from 0.9, decay 0.1/300 steps) with ILT w/o scheduling at constant p=0.3. These conditions differ in two ways: the presence/absence of decay and the average fraction of text tokens seen during training. The paper never reports the scheduled run's time-averaged effective text ratio; moreover, footnote 5 states that Algorithm 1's stopping rule makes the actual ratio larger than p, and the Poisson lambda is unreported. A constant p=0.3 control therefore cannot establish that gradual decay, rather than total text exposure, drives the gains. Since Section 3.2 identifies the scheduling as 'the main difference from SpiRitLM,' and the abstract attributes the improvements to better modality adaptation, this confound directly undermines the paper's central new claim. Additional ablations over schedule parameters (initial p, decay step) are also absent, so the specific schedule is not validated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes scheduled interleaved speech-text training for speech-to-speech translation (S2ST) with LLMs. The method interleaves word-aligned text tokens into speech-unit sequences at both the source and target sides, and gradually decays the text ratio from 0.9 to 0 during fine-tuning. Experiments on CVSS with LLaMA3.2-1B show consistent ASR-BLEU improvements over a speech-only baseline across seven languages, with larger gains in lower-resource languages. Ablations indicate that both-side interleaving, real CTC word alignments, and interleaving rather than masking are all important. The paper also analyzes the evolution of length and representation gaps during training.","tokens_in":9701,"tokens_out":6597,"duration_ms":78272,"significance":"If the central claim holds, the method provides a straightforward and effective way to adapt text-only LLMs to S2ST, which is particularly valuable for low-resource language pairs. The ablation design is thoughtful: it separates the contributions of input/output interleaving, alignment quality, and the difference between interleaving and masking. The analysis in Figure 3 offers a plausible mechanistic explanation through progressive modality adaptation. However, the specific claim that the decay schedule itself is beneficial is not yet empirically isolated, because the only non-scheduled control is confounded with average text ratio. The paper is otherwise well positioned for the speech/LLM community, and the identified issue is fixable with additional controlled experiments.","major_comments":[{"comment":"The comparison that is supposed to isolate the effect of scheduling compares 'Scheduled ILT' (p decaying from 0.9 by 0.1 per 300 steps) with 'ILT w/o scheduling' at a constant p=0.3. These two conditions differ in two ways: the presence/absence of decay and the average fraction of text tokens observed during training. The scheduled run's time-averaged text ratio is never reported, and footnote 5 states that the actual ratio exceeds p because of Algorithm 1's stopping rule, with the Poisson span-length parameter lambda left unreported. This means the constant-p=0.3 control cannot establish that gradual decay, rather than the total quantity of text exposure, drives the gains. The confound is compounded by the fact that constant p=0.3 is worse than the baseline in Fr-en, De-en, Es-en, and Ca-en (25.8 vs 28.8, 22.1 vs 27.3, 21.9 vs 33.5, 17.8 vs 23.8), so it is not a neutral reference. To support the central claim about scheduling, the authors should add a constant-p condition matched to the scheduled run's time-averaged actual text ratio, and ideally also report schedule ablations (different initial p, different decay steps, or an increasing-p schedule) to show that the direction and rate of decay matter.","section":"Section 3.2 and Table 2"},{"comment":"No multiple seeds, significance tests, or confidence intervals are reported for any condition. The paper's central empirical claims are that scheduled ILT consistently improves over baseline and that scheduling is effective. The cross-language trend is consistent, but all results are single runs; observed differences of 2-8 ASR-BLEU points could be affected by training variability, especially for the smaller low-resource subsets. At least 2-3 seeds with means and standard deviations, or pairwise significance tests, should be provided for the key comparisons in Tables 2 and 3.","section":"Tables 2 and 3"}],"minor_comments":[{"comment":"The Poisson parameter lambda used to sample span lengths in Algorithm 1 is never reported, even though it directly affects the realized text ratio and therefore the quantities compared in Table 2. Please report lambda and the empirical average text ratio for each condition.","section":"Section 4 / Algorithm 1"},{"comment":"The total number of training steps per language is not reported, so the meaning of 'every 300 steps' for the decay schedule is unclear. Please report the number of steps or epochs for each language subset.","section":"Section 4"},{"comment":"The ablation 'Interleave without alignment' shows that using equal-interval spans degrades performance (11.5 vs 19.5 ASR-BLEU on Pt-en), but the quality of the CTC alignments for the seven languages is not assessed. Reporting word/character alignment error rates, at least for the low-resource languages, would make the reliance on alignment more transparent.","section":"Section 4 / Table 3"},{"comment":"The representation-similarity analysis in Figure 3 is only shown for Pt-en and is described qualitatively. Reporting the numerical similarity values at the final training step, or a summary statistic, would strengthen the claim that scheduled ILT leads to better final speech-text alignment.","section":"Figure 3"},{"comment":"The phrase 'especially for languages with limited training data' is supported by a visual trend in Table 2, but no correlation coefficient or other statistical evidence is provided. A simple linear regression of gains against training hours, or a scatter plot, would substantiate this claim.","section":"Abstract and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the experiments are clean, but the main novelty—scheduled decay—is not isolated from the average text ratio. This is a fixable issue that likely requires new experiments rather than reanalysis. If the authors can add a matched constant-p control and/or schedule ablations, the paper could be acceptable. I would also encourage the editor to ask for multiple seeds, as the reported differences are not accompanied by variance estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent empirical study of interleaved speech-text fine-tuning for LLM-based speech-to-speech translation. The consistent ASR-BLEU gains across all seven CVSS languages are real, and the ablations are informative. But the paper's central new claim—that the decaying schedule itself helps—rests on a comparison that is confounded, and the stress-test note is right to flag it.\n\nScheduled ILT starts at p=0.9 and decays by 0.1 every 300 steps; the non-scheduled control uses a constant p=0.3. Those conditions differ in both the presence of decay and the average text ratio seen during training. Footnote 5 says the effective ratio is larger than p, and lambda is unreported, so you cannot reconstruct the scheduled run's time-averaged exposure. Without a constant-p condition matched to that average (or a small sweep of constants), the contrast cannot isolate scheduling. This matters because the paper explicitly frames scheduling as the main difference from SpiRitLM, and the abstract credits the improvements to progressive modality adaptation.\n\nThe paper does several things well. It credits SpiRitLM for word-level interleaving and positions its contribution at fine-tuning time. Algorithm 1 is concrete, and using CTC alignments from an ASR-fine-tuned w2v-BERT avoids external aligners. Table 3's ablations are genuinely useful: both-side interleaving, real word alignment, and text replacement over masking all matter. The low-resource gains (Pt-en from 10.3 to 19.5, Ru-en from 6.0 to 14.1 ASR-BLEU) are large and consistent with the narrative that interleaving helps when supervised data is scarce.\n\nThe soft spots are real. No significance tests or multiple seeds, a 1K test subsample, and an unablated hand-picked schedule. No code, configs, or lambda. The dependence on CTC alignment is a plausible fragility, but the consistent gains suggest it is not fatal. These issues make the paper conditional, not a reject.\n\nWho gets value? Researchers and practitioners fine-tuning LLMs for speech tasks, especially low-resource S2ST. It deserves a serious referee. A revision that adds a matched constant-p control, multi-seed significance, schedule ablations, and released configs would move it to accept; without the control, the scheduling claim is not established.","headline":"Useful interleaved fine-tuning recipe, but the scheduling claim is not supported by the only comparison that tests it.","tokens_in":10438,"tokens_out":3558,"would_cite":false,"duration_ms":40831,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning an LLM for speech-to-speech translation is more effective when word-aligned text tokens are interleaved into the speech-unit sequence and their ratio is gradually decayed to zero during training.","keywords":["speech-to-speech translation","large language model","modality adaptation","interleaved speech-text training","discrete speech units","CTC alignment","curriculum learning","multilingual translation"],"falsifier":"Take a low-resource language pair and rerun scheduled interleaving with alignments deliberately corrupted (e.g., random shifts of span boundaries or equal-interval spans). If the ASR-BLEU gain over the speech-only baseline does not disappear or shrink substantially compared to the aligned version, then alignment accuracy is not the load-bearing condition the paper claims.","tokens_in":9256,"feed_emoji":"🎙️","tokens_out":4777,"duration_ms":41951,"temperature":0.7,"pith_summary":"The paper tries to establish that a pre-trained text LLM can be adapted to speech-to-speech translation more reliably if, during fine-tuning, its training sequences mix discrete speech units with word-aligned text tokens, with the text ratio decaying from 0.9 to 0 as training proceeds. The authors argue that this scheduled interleaving eases the modality gap between text and speech, and they report consistent ASR-BLEU gains over speech-only fine-tuning across all seven CVSS language pairs tested. The gains are largest for the lowest-resource pairs (Portuguese-English and Russian-English), where translation quality roughly doubled. If the claim holds, the method offers a simple, alignment-based recipe for squeezing better S2ST performance out of limited supervised data without changing the architecture.","feed_headline":"Scheduled text-speech interleaving lifts S2ST BLEU by up to 9 points","feed_subtitle":"Fine-tuning LLaMA with word-aligned text that fades out improves translation, especially for low-resource languages.","key_machinery":"The central mechanism is Algorithm 1, which constructs interleaved speech-text units by randomly selecting word spans (span lengths drawn from a Poisson distribution) and replacing the corresponding speech-unit spans with their BPE text tokens. The replacement is governed by a text ratio p, which the paper schedules to decay from 0.9 by 0.1 every 300 training steps, so the model starts with mostly text-like input and gradually shifts to pure speech units. Word alignments come from the CTC output of an ASR-fine-tuned w2v-BERT encoder, so no external aligner is needed. This machinery is doing the work of progressively bridging the length and representation gaps between speech and text modalities during fine-tuning.","core_discovery":"The central claim, stated in the abstract and conclusion, is that scheduled interleaved speech-text training improves speech-to-speech translation performance of LLMs, attributable to better modality adaptation. Concretely, fine-tuning LLaMA3.2-1B with interleaved speech-text units whose text ratio decays from 0.9 to 0 outperforms fine-tuning on speech units alone in ASR-BLEU on all seven CVSS pairs, e.g. Pt-en 19.5 vs 10.3 and Ru-en 14.1 vs 6.0, while UTMOS speech quality stays roughly constant. The paper further claims the scheduling itself matters: interleaving at a constant text ratio of 0.3 is worse, and interleaving on both the input and output sides, guided by word-level CTC alignments, is necessary.","pith_inferences":["The method's benefit should transfer to other decoder-only speech tasks, such as spoken dialog or speech continuation, wherever a text-pretrained model must be adapted to discrete speech units with limited data.","The schedule hyperparameters (starting ratio 0.9, decay step 300, Poisson span length) are untested across tasks; the optimal decay may depend on language pair length ratios and data size, so the method may need per-task tuning.","Because the gain concentrates in low-resource pairs, a testable extension is to use the scheduled interleaving as the fine-tuning objective for zero-shot or few-shot S2ST with more languages than seen in supervised data."],"forward_implications":["Scheduled ILT improves ASR-BLEU over the speech-only baseline on all seven tested CVSS language pairs, with the largest gains on low-resource pairs.","Interleaving at a constant text ratio does not reproduce the gain; the decay schedule is a necessary ingredient.","Both source-side and target-side interleaving are needed; interleaving only one side loses most of the benefit.","Accurate word-level alignment is required; replacing speech spans at equal intervals instead of at CTC alignments drops Pt-en from 19.5 to 11.5 ASR-BLEU.","Speech quality, measured by UTMOS, stays high (around 4.2) across training methods, so the translation gain is not bought with degraded audio."],"supporting_citations":[{"why":"Supplies the AudioPaLM architecture and fine-tuning approach for LLM-based S2ST that this paper builds on.","marker":"[7]"},{"why":"Introduces interleaved speech-text representations, the idea this paper adapts to fine-tuning with a decay schedule.","marker":"[25]"},{"why":"Provides the w2v-BERT self-supervised encoder from which semantic speech units are extracted.","marker":"[11]"},{"why":"Supplies the CTC-segmentation method used to obtain word-level alignments for the interleaving algorithm.","marker":"[35]"},{"why":"Provides the CVSS multilingual S2ST corpus used for training and evaluation.","marker":"[27]"},{"why":"The LLaMA3.2-1B text LLM that is fine-tuned in the experiments.","marker":"[26]"},{"why":"Shows speech resynthesis from discrete units, the basis for the unit HiFi-GAN vocoder used to generate waveforms.","marker":"[28]"},{"why":"Provides the CoVoST 2 corpus used for ASR fine-tuning of the encoder and for building CVSS.","marker":"[36]"}],"fun_headline_variants":["Scheduled text fade improves LLM speech-to-speech BLEU","Interleaving text then speech lifts S2ST scores","Low-resource speech translation boosted by scheduled text","LLMs translate speech better with decaying text ratio","Word-level interleaving key to better speech translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the CTC word alignments correctly identify, for every training utterance in every source language, which spans of speech units correspond to which words.","fun_headline_variants_meta":{"raw":{"variants":["Scheduled text fade improves LLM speech-to-speech BLEU","Interleaving text then speech lifts S2ST scores","Low-resource speech translation boosted by scheduled text","LLMs translate speech better with decaying text ratio","Word-level interleaving key to better speech translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1428,"prompt_tokens":892,"completion_tokens":536,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":457}},"tokens_in":508,"tokens_out":536,"duration_ms":6444,"temperature":1.0,"reasoning_tokens":457,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:30:52.757421+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a low-resource language pair and rerun scheduled interleaving with alignments deliberately corrupted (e.g., random shifts of span boundaries or equal-interval spans). If the ASR-BLEU gain over the speech-only baseline does not disappear or shrink substantially compared to the aligned version, then alignment accuracy is not the load-bearing condition the paper claims.","supporting_citations":[{"cited_title":"The ATR multilingual speech-to-speech translation system,","cited_arxiv_id":null,"evidence_quote":"Supplies the AudioPaLM architecture and fine-tuning approach for LLM-based S2ST that this paper builds on."},{"cited_title":"Qwen-Audio: Advancing universal audio understanding via unified large-scale audio-language models,","cited_arxiv_id":null,"evidence_quote":"Introduces interleaved speech-text representations, the idea this paper adapts to fine-tuning with a decay schedule."},{"cited_title":"UnitY: Two- pass direct speech-to-speech translation with discrete units,","cited_arxiv_id":null,"evidence_quote":"Provides the w2v-BERT self-supervised encoder from which semantic speech units are extracted."},{"cited_title":"SALMONN: Towards generic hearing abilities for large language models,","cited_arxiv_id":null,"evidence_quote":"Provides the CVSS multilingual S2ST corpus used for training and evaluation."},{"cited_title":"SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities,","cited_arxiv_id":null,"evidence_quote":"The LLaMA3.2-1B text LLM that is fine-tuned in the experiments."},{"cited_title":"PaLM 2 technical report,","cited_arxiv_id":null,"evidence_quote":"Shows speech resynthesis from discrete units, the basis for the unit HiFi-GAN vocoder used to generate waveforms."},{"cited_title":"On generative spoken language modeling from raw audio,","cited_arxiv_id":null,"evidence_quote":"Provides the CoVoST 2 corpus used for ASR fine-tuning of the encoder and for building CVSS."}],"review_version":1}