{"id":"9b5d0349-957a-488e-8434-70862e83d465","arxiv_id":"2602.11933","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning speech translation on adversarial text embeddings in an aligned speech-text space transfers inflectional robustness to speech: ~3 BLEU average gain on adversarially inflected audio, with no adversarial speech used in training.","lead":"This paper shows speech-translation models break on small word-form changes common in non-native speech, then hardens them by training on adversarial text edits injected into the model's shared speech-text space — no fake audio needed. The recipe lifts translation quality by ~3 BLEU points on adversarially inflected audio across four language pairs, with only a small dip on clean speech.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CMRT's central mechanism is unverified: no direct measurement that injected adversarial text embeddings approximate test-time speech-encoder embeddings of MORPHEUS TTS, so the robust gains could come from KL/mixup regularization rather than genuine cross-modal transfer.","rationale":"The reader's weakest_assumption correctly identifies the embedding-gap as the load-bearing premise. The central claim is not just that CMRT-FN improves robustness, but that it does so by simulating adversarial speech perturbations through injected adversarial text embeddings in an aligned space. This causal mechanism is supported only by correlation (Figure 3) and by representational similarity (CKA), neither of which rules out the simpler explanation that the fine-tuning procedure—especially the asymmetric KL regularization—improves robustness regardless of the adversarial content. The §5.3 plateau is direct evidence that the gap is nontrivial, but the paper does not quantify it or test whether the specific adversarial embeddings are necessary. The proposed clean-text ablation and embedding-distance measurement would settle this. I do not think this concern warrants rejection: the empirical gains are consistent across four language pairs and two model scales, and even if the mechanism is weaker than claimed, the method may still be a useful robustness technique. The reader's other concerns (no significance tests, missing baselines, limited attack scope) are real but secondary; the mechanism question is more fundamental because it determines whether the paper's central conceptual contribution is valid. Thus the verdict remains CONDITIONAL, unchanged from the reader's assessment.","tokens_in":15254,"tokens_out":9319,"duration_ms":93320,"concrete_test":"Run CMRT-FN identically (same data, λkl=5, frozen speech encoder) but replace the adversarial text embeddings for the attacked words with the corresponding clean text embeddings; report MORPHEUS BLEU. Additionally, on a MORPHEUS dev set, measure cosine similarity between the frozen speech encoder's mean-pooled embeddings of adversarial TTS word segments and (i) the inflected word's text embedding, (ii) the clean word's text embedding. If the clean-text injection retains most of the robustness gain over CMRT-TR, or if (i) is not greater than (ii), the central cross-modal transfer mechanism is unsupported and the gains are better attributed to fine-tuning/regularization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is the embedding-gap assumption in §3.2.3: that for words affected by MORPHEUS, the frozen HuBERT/mHuBERT speech encoder's test-time embeddings of adversarial TTS speech lie close to the adversarial text embeddings injected during CMRT-FN. This is never measured. During fine-tuning the model sees clean speech frames a, with the attacked words replaced by text embeddings of inflected forms; at test it sees TTS audio of the inflected transcription through the same frozen encoder. If the encoder's representation of that audio is not close to the inflected word's text embedding, the training signal is mismatched with the test-time input. §5.3 shows CMRT-FN plateaus 1.4 BLEU below TTS-Morpheus-FN and attributes this to 'modality specific information that limits cross-modal transfer'—an explicit admission that the assumption holds only partially. The only support for the mechanism is the correlational Figure 3 and the CKA analysis, neither of which demonstrates that the injected text embeddings are the operative factor. A simpler explanation is that the asymmetric KL losses (Eq. 16-17) plus fine-tuning act as regularization that helps MORPHEUS inputs generally; without a control in which non-adversarial text embeddings are injected for the attacked words, the claimed text-to-speech 'simulation' is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Cross-Modal Robustness Transfer (CMRT), a two-stage method for improving the robustness of end-to-end speech translation (E2E-ST) models to inflectional morphological perturbations without using adversarial speech during training. In the first stage (CMRT-TR), the model is trained to align speech and text representations via word-aligned contrastive learning (WACO), mixup training, and symmetric KL divergence. In the second stage (CMRT-FN), the speech encoder is frozen and the translation encoder-decoder is fine-tuned on \"adversarial mixup\" sequences in which the embeddings of words attacked by the MORPHEUS inflection generator are replaced by their adversarial text embeddings, with asymmetric KL losses to clean speech/text outputs. The paper also introduces Speech-MORPHEUS, a TTS-based adaptation of MORPHEUS, and reports experiments on CoVoST 2 (En-De, En-Ca, En-Ar, Fr-En) showing average gains of 3.4 BLEU over HuBERT-Transformer and 3.1 over CMRT-TR on the adversarial test set, while falling 1.4 BLEU short of TTS-side adversarial fine-tuning, and a smaller clean-set accuracy drop. An appendix extends the results to NLLB-initialized models with an average gain of 4.4 BLEU.","tokens_in":15558,"tokens_out":6105,"duration_ms":62253,"significance":"If the central claim holds, the paper would make a useful practical contribution: it addresses a real and under-studied failure mode of E2E-ST models (morphological robustness), and it proposes a comparatively cheap method that avoids large-scale adversarial TTS generation. The empirical scope is reasonable—four language pairs, a strong TTS-side adversarial training baseline, a clean-TTS control, and an appendix with a massively pre-trained initialization. The paper is also honest about the robustness-accuracy trade-off and about the plateau relative to speech-side fine-tuning in Section 5.3. However, the current evidence does not yet establish the proposed mechanism. The training and test perturbations come from the same MORPHEUS generator, the central embedding-gap assumption is never directly measured, and the design lacks a control that isolates adversarial text substitution from the regularizing effect of the KL losses. These issues are fixable with additional experiments and analysis, but they are load-bearing for the paper's key claims.","major_comments":[{"comment":"The headline results are point estimates from single runs, with no variance, confidence intervals, or significance testing. For example, the En-Ar Morpheus column reports 14.5 for CMRT-FN versus 12.7 for CMRT-TR, and the Fr-En difference is 24.6 versus 21.9; these differences may fall within normal test-set noise for ST BLEU. Since the main contribution is quantitative (average +3.4/+3.1, and +4.4 in Appendix B), please report multiple seeds with means and standard deviations, or at least paired bootstrap confidence intervals over test segments, and state whether the per-direction and average gains are statistically significant.","section":"§5.1, Table 1"},{"comment":"There is a distribution-matching concern: the training-time adversarial text is produced by the MORPHEUS generator, and the Speech-MORPHEUS test set is produced by the same generator's inflections rendered with XTTS-v2. The measured gains could therefore reflect adaptation to the specific inflection inventory and selection behavior of MORPHEUS rather than general morphological robustness. To support the claim of robustness transfer, the evaluation should include held-out perturbation types (e.g., a different inflector, a different POS inventory, or a held-out subset of inflections) and TTS voices/speakers not used in development. This is a load-bearing issue because the paper's stated goal is robustness to real non-native/dialectal morphological variation, not just one attack generator.","section":"§3.3, §5.1"},{"comment":"The core mechanism is unverified. The paper claims that injecting adversarial text embeddings into the speech manifold simulates adversarial speech perturbations, which requires that the frozen speech encoder's test-time embeddings of TTS-rendered inflected words lie close to the text embeddings used during CMRT-FN. This is never directly measured. Figure 3 is correlational (alignment strength is confounded with training objective), and the CKA analysis in §5.4 uses whole-sentence mean representations, which dilutes the effect of attacked words. Moreover, §5.3 explicitly reports a plateau below TTS-Morpheus-FN and attributes it to 'modality specific information that limits cross-modal transfer'—an admission that the assumption holds only partially. Please measure the cosine distance between text embeddings of inflected forms and speech-encoder embeddings of the corresponding XTTS audio (","section":"§3.2.3, Eq. (14), §5.3"},{"comment":"The gains attributed to adversarial text substitution could instead come from the asymmetric KL losses (Eqs. 16-17), which resemble VAT-style regularization, or from additional fine-tuning on the clean ST/MT objectives. The paper lacks a control that isolates the adversarial substitution component. Please add ablations in which (a) the same loss is used but clean text embeddings are inserted at attacked positions, (b) random non-adversarial text substitutions are used at attacked positions, and (c) CMRT-TR is fine-tuned with only the clean ST/MT and KL losses. If (a) or (b) recovers most of the gain, then the claim that adversarial text embeddings are the operative factor is not supported.","section":"Eqs. (15)-(18), §5.2"}],"minor_comments":[{"comment":"Typo: 'highly vulnerable it' should be 'highly vulnerable to it'.","section":"Abstract"},{"comment":"The difference between 'CMRT-FN (50K)' and 'CMRT-FN' is not explained in the text. Please clarify whether both use 50K adversarial text examples, and if not, what dataset size the second variant uses and why it is reported alongside the 50K variant.","section":"Table 1"},{"comment":"Please add axis labels, specify which BLEU score is plotted (adversarial test set), include error bars or a confidence interval for the cosine similarity, and report the correlation coefficient and its significance. The current caption is too underspecified for the strong qualitative claim.","section":"Figure 3"},{"comment":"Typo: 'On overage' should be 'On average'.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem and the basic idea is plausible, but the current evidence is not yet convincing. The requested ablations and direct embedding-gap measurement are necessary before the central mechanism claim can be accepted. I would not reject on novelty grounds; the proposed method is a sensible next step following WACO/STEMM-style alignment work. One additional concern for reproducibility: no code or configuration files are provided, and the method has several interacting components; a release would materially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: worth a look before the next deadline. CMRT — fine-tuning an end-to-end speech translation model by injecting adversarial text embeddings into a speech-text aligned space — gives a real, consistent robustness gain across four directions and two model scales, without using adversarial speech for training. The headline number holds up: average +3.4 BLEU over HuBERT-Transformer on the Speech-MORPHEUS test set, and +4.4 in the NLLB appendix. The clean-test drop is small (0.6–0.7) compared with the 3.6 drop from TTS-side adversarial fine-tuning.\n\nWhat is actually new: the recipe of substituting adversarial text embeddings for attacked words during mixup-style fine-tuning, with a frozen speech encoder and asymmetric KL, is not in the cited WACO, STEMM, CMOT, or MORPHEUS work. The CKA analysis and the cosine-similarity correlation are sensible attempts to show that CMRT-FN moves the model toward the same solution as speech-side adversarial fine-tuning.\n\nThe soft spots are real but mostly addressable. The load-bearing claim — that adversarial text embeddings sit close enough to the speech encoder's TTS rendering of the inflected word to act as a faithful substitute — is never directly tested. The paper shows correlation and representation similarity, but no measurement of the embedding gap itself. Figure 3 and the CKA results are consistent with the mechanism; they don't nail it. §5.3 honestly acknowledges the plateau below TTS-Morpheus-FN and attributes it to 'modality specific information that limits cross-modal transfer,' which is an admission that the substitution works only partially. Because there is no control that injects non-adversarial text embeddings for the same attacked words, part of the gain might just be the KL/mixup regularization doing the work. That is the biggest missing experiment, and it is cheap to run.\n\nAlso worth flagging: all BLEU scores are single runs, with no variance or significance testing; the adversarial data-generation pipelines for Catalan and Arabic are not described, despite covering three of four directions; and since MORPHEUS generates both the training and test perturbations, the claim is about robustness to this specific generator, not general robustness. The 'overcoming the robustness-accuracy trade-off' phrasing oversells a smaller drop, not an absence of a drop.\n\nOverall: the empirical core is consistent and the idea is genuinely useful for the E2E-ST robustness literature. I would send it to serious peer review, conditional on error bars, the non-adversarial control, and pipeline details for all languages.","headline":"Solid empirical paper with a genuinely new robustness-transfer idea; the 3+ BLEU gain is consistent, but the mechanism is under-validated and the evaluation lacks variance and a non-adversarial control.","tokens_in":16132,"tokens_out":3073,"would_cite":true,"duration_ms":29103,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on adversarial text embeddings—injected into an aligned speech-text embedding space—improves speech translation robustness to inflectional errors by over 3 BLEU points across four language pairs, with no adversarial speech neede","keywords":["cross-modal robustness transfer","speech translation","adversarial robustness","morphological inflection","contrastive learning","mixup training","Speech-MORPHEUS","robustness-accuracy trade-off"],"falsifier":"Measure the cosine distance between the adversarial text embeddings used in CMRT-FN and the actual speech-encoder embeddings of the same inflected words produced by the TTS system. If the distance is systematically large, or if fine-tuning with random (non-adversarial) text embeddings yields the same BLEU gain on Speech-MORPHEUS, the claimed cross-modal transfer mechanism is not what delivers the robustness.","tokens_in":15078,"feed_emoji":"🎧","tokens_out":9015,"duration_ms":139639,"temperature":0.7,"pith_summary":"The paper claims that acoustic robustness—here, resilience to inflectional errors typical of non-native or dialectal speech—can be transferred from the text modality to the speech modality without ever generating adversarial audio. The method first trains a speech-to-text translation model with word-level contrastive learning and mixup training so that speech and text embeddings of the same word occupy a shared semantic space. Then it fine-tunes the model by swapping in adversarial text embeddings for inflected words, keeping the speech encoder frozen. Across English-to-German/Catalan/Arabic and French-to-English, this raises BLEU on the adversarial spoken test set by more than 3 points over a strong baseline, reaching within about 1.4 BLEU of direct TTS-side adversarial fine-tuning while losing far less on clean speech. The paper's own plateau analysis shows the transfer is partial, and the limitations section acknowledges a slight clean-set drop on English-to-X directions and the extra training stage—so the robustness gain is not free.","feed_headline":"Adversarial text alone lifts speech translation robustness","feed_subtitle":"Injecting adversarial text embeddings transfers inflection robustness to speech, cutting the need for synthetic audio.","key_machinery":"The load-bearing mechanism is the aligned speech-text embedding space built by CMRT-TR, combined with an adversarial-mixup fine-tuning step (CMRT-FN). Word-aligned Contrastive Learning (WACO) pulls speech and text embeddings of the same word together; mixup training and symmetric KL-divergence familiarise the translation encoder-decoder with mixed representations. In the robustness stage, the frozen speech encoder's output for attacked words is replaced by adversarial text embeddings (the inflected form), and an asymmetric KL loss pushes the model's distribution on that adversarial mixup toward its distributions on clean speech and text. The paper also constructs the evaluation tool, Speech-","core_discovery":"On the paper's own terms, the central discovery is that morphological robustness transfers across modalities through an aligned embedding space. Using a combination of word-aligned contrastive learning, mixup training, and symmetric KL regularisation, the authors construct a model (CMRT-TR) whose speech and text representations for the same word are close. Fine-tuning this model (CMRT-FN) with adversarial text embeddings—keeping the speech encoder frozen and using an asymmetric KL to keep the adversarial mixup's output distribution close to that of clean inputs—produces a model that, on the Speech-MORPHEUS adversarial test set, gains an average of 3.4 BLEU over HuBERT-Transformer and 3.1 ove","pith_inferences":["If the mechanism is genuinely cross-modal, the same trick should transfer other text-side corruptions (typos, dialectal lexicon) to speech without audio-side data — a testable extension beyond inflectional morphology.","The plateau below TTS-side fine-tuning (§5.3) suggests that contrastive alignment leaves residue of modality-specific information; methods that more aggressively minimise the modality gap might close the remaining ~1.4 BLEU.","A control experiment replacing adversarial text embeddings with arbitrary inflected text embeddings would determine whether the gains come from adversarial targeting or merely from mixup regularisation; the paper does not report this control.","Since Speech-MORPHEUS is TTS-generated, part of the measured robustness may be an artefact of TTS normalisation; the 'Clean' column of Table 1 provides a partial baseline for this confound."],"forward_implications":["Speech translation systems can be hardened against a realistic class of morphological variation at a fraction of the compute cost of synthetic adversarial audio.","The robustness–accuracy trade-off is partially decoupled: CMRT-FN's clean-set loss (~0.6 BLEU) is much smaller than the 3.6 BLEU loss of TTS-side adversarial fine-tuning.","The transfer mechanism scales to massively pre-trained translation backbones (NLLB-initialised models), where CMRT-FN gains ~4.4 BLEU on adversarial speech, suggesting the method is not tied to a small architecture.","Word-level alignment quality becomes a measurable driver of adversarial robustness, giving representation-learning objectives a concrete safety payoff.","Because the method requires no TTS at training time, it can be applied to language pairs where high-fidelity TTS does not exist."],"fun_headline_variants":["Text-only attacks harden speech translation","Cross-modal trick: text boosts speech robustness","No adversarial audio needed: text alone hardens ST","Text-to-speech robustness transfer: 3+ BLEU gain","Adversarial text only: speech translation gets tough"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole method rests on the assumption that the frozen speech encoder, at test time, produces embeddings for genuinely inflected speech that lie close enough to the adversarial text embeddings used in fine-tuning that the text substitution is equivalent to training on the real acoustic perturbation; the paper's own plateau result shows this holds only partially, and if the gap is large the measured gains reflect matching to the MORPHEUS text distribution rather than true cr","fun_headline_variants_meta":{"raw":{"variants":["Text-only attacks harden speech translation","Cross-modal trick: text boosts speech robustness","No adversarial audio needed: text alone hardens ST","Text-to-speech robustness transfer: 3+ BLEU gain","Adversarial text only: speech translation gets tough"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000915,"raw_usage":{"total_tokens":3750,"prompt_tokens":713,"completion_tokens":3037,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":2962}},"tokens_in":457,"tokens_out":3037,"duration_ms":25617,"temperature":1.0,"reasoning_tokens":2962,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:56:11.447799+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the cosine distance between the adversarial text embeddings used in CMRT-FN and the actual speech-encoder embeddings of the same inflected words produced by the TTS system. If the distance is systematically large, or if fine-tuning with random (non-adversarial) text embeddings yields the same BLEU gain on Speech-MORPHEUS, the claimed cross-modal transfer mechanism is not what delivers the robustness.","supporting_citations":[],"review_version":1}