Pith. sign in

REVIEW 4 major objections 6 minor 67 references

Representation Purification for End-to-End Speech Translation

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A linear projection that removes timbre, pitch, rhythm, and noise from speech representations improves end-to-end translation on every tested language direction.

desk verdict Consistent BLEU gains across many settings, but the purification mechanism is unproven; the content-leakage concern is real and should be addressed. read the letter →

arxiv 2412.04266 v1 pith:OKPIKTRI submitted 2024-12-05 cs.CL cs.SDeess.AS

Chengwei Zhang , Yue Zhou , Rui Zhao , Yidong Chen , Xiaodong Shi This is my paper · ORCID
classification cs.CLcs.SDeess.AS
keywords speech-to-texttranslationrepresentationpurificationorthogonalprojectioncontent-agnosticinformationend-to-endspeechdisentanglementmutualminimizationtranscript-freetraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speech-to-text translation systems must transcribe meaning while carrying the speaker's voice, and this paper argues that the non-linguistic residue—timbre, pitch, rhythm, background noise—actively hurts translation. It models a speech representation as content-relevant information plus content-agnostic information, and proposes SRPSE, which learns a content-agnostic encoder, projects the full representation onto that encoder's direction, and subtracts the projection so the decoder sees a purified representation. On MuST-C and CoVoST-2, across ten translation directions and three training settings, the paper reports higher BLEU scores than its baselines in every direction, with the largest advantage when no transcription is used. The result matters because transcript-free speech translation is the setting that works for unwritten languages, where a purification that needs no annotations has direct practical value.

What carries the argument

The load-bearing object is the orthogonal projection purification (OPP) module. It takes a content-agnostic representation $H_\alpha$ and a complex-information representation $H_\beta$, projects $H_\beta$ onto the $H_\alpha$ direction, and subtracts that projection to obtain the purified representation $H_\gamma$; the same projection also feeds a variational mutual-information upper bound (vCLUB) that discourages residual overlap between $H_\gamma$ and $H_\beta^*$. Around the projection, a supervision-enhancement scheme perturbs speech with noise, pitch shift, and time stretch, then uses speaker-ID and SNR classifier losses plus a consistency loss to make $H_\alpha$ capture content-agnostic factors without needing transcriptions.

What would settle it

Take two recordings of the same sentence spoken with very different voices and noise floors, run both through SRPSE, and measure both how far apart the purified representations are and how close the decoded translations are. If the purified representations stay as far apart as the raw ones, or if removing the learned content-agnostic direction degrades a speech-recognition probe trained on the purified representation, the projection is not actually isolating content-agnostic information.

Watch

Extended reading notes

Core claim

The central claim is that content-agnostic speech factors occupy a usable direction in the learned representation space, and that subtracting that direction gives a cleaner input for translation. Given a content-agnostic representation $H_\alpha$ from a dedicated encoder and a complex representation $H_\beta$, SRPSE computes the projection $H_\beta^* = (H_\beta \cdot H_\alpha / |H_\alpha|)(H_\alpha/|H_\alpha|)$ and keeps $H_\gamma = H_\beta - H_\beta^*$. It supervises $H_\alpha$ with speaker and signal-to-noise classifiers, minimizes a variational upper bound on the mutual information between $H_\gamma$ and $H_\beta^*$, and uses a consistency loss between purified representations of original and perturbed speech. The paper reports that this purification improves BLEU on all ten tested language pairs, raises speech-text representation similarity from 0.32 to 0.58, and reduces the translation gap between original and voice-converted audio.

Load-bearing premise

The method assumes that the non-linguistic parts of speech—voice, pitch, rhythm, noise—live in a separate direction in the representation space, so subtracting that direction does not remove any of the words.

Editorial extensions

If this is right

  • Transcript-free speech translation improves on all eight MuST-C directions and both CoVoST-2 directions, so the purification is usable where transcriptions are unavailable.
  • Purification also improves multi-task and expanded settings, meaning it composes with machine-translation knowledge transfer rather than replacing it.
  • Purified representations are closer to text representations (cosine similarity 0.58 vs 0.32) and stay closer when the audio is voice-converted, implying more stable decoding.
  • The added modules increase inference time by about 5 percent, so the purification does not demand a large compute budget.
  • Because the method needs no transcriptions, it can be applied to speech-plus-translation data for unwritten or low-resource languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same linear subtraction should generalize to other speech tasks where non-content factors interfere, such as ASR or speaker-independent speech understanding; this paper only tests translation, so that extension is speculation.
  • Inference: the content-agnostic subspace is learned from speaker IDs and SNR labels present in the training data; unseen nuisance sources such as room reverberation, channel effects, or emotion may not lie in that subspace, and the paper's own limitations section acknowledges that many content-agnostic factors remain unexplored.
  • Inference: a direct test would be to train a speech-recognition probe on $H_\gamma$: if word-error rate stays close to the raw representation while speaker identification accuracy drops, the purification is genuinely content-preserving; otherwise it is trading one nuisance for another.
  • Inference: the architecture, not the augmented data, is what drives the gain; the paper's comparison with plain augmentation supports this, and it suggests that adding more perturbations to the same projection may keep improving robustness without retraining the projection.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SRPSE, a framework for end-to-end speech translation that attempts to purify speech representations by removing content-agnostic information. It adds a content-agnostic encoder (CA-Enc) trained with speaker-ID and SNR classification losses, a complex-information encoder (CI-Enc), and an orthogonal projection purification (OPP) module that subtracts the CA-Enc component from the CI-Enc representation. Training also includes a consistency loss between original and perturbed speech inputs and a variational mutual-information upper-bound loss. The method is evaluated on MuST-C (eight En-to-X directions) and CoVoST-2 (two X-to-En directions) under transcript-free, multi-task, and expanded settings, reporting BLEU improvements over several strong baselines, with additional ablations and analyses of robustness, augmentation, and inference cost.

Significance. If the results hold, the paper makes a useful contribution to end-to-end speech translation by directly addressing non-content speech factors and by demonstrating consistent gains across ten translation directions and three training settings. The experimental design has notable strengths: ablations isolate each training objective, a data-augmentation baseline is included to distinguish the proposed mechanism from simple input perturbation, the computational overhead is measured, and comparisons cover several recent strong baselines. However, the central mechanistic claim—that the gains come from removing content-agnostic information rather than from some other property of the architecture—is not fully established because no training signal guarantees that the content-agnostic encoder is free of lexical content. In addition, the statistical significance claims are not supported by a described test or variance estimates. The paper is therefore a promising systems contribution whose main interpretation needs additional evidence.

major comments (4)
  1. [§3.2, Eqs. (5)-(6)] The orthogonal projection operation is underspecified. In Eqs. (5)-(6), Hα and Hβ are treated as vectors with a dot product and a norm, but earlier in the same section they are described as encoder output representations with a temporal dimension; Eq. (7) even indexes them by time step t. The paper never states whether the projection is applied per time step, to mean-pooled sentence vectors, or through learned linear maps. Since OPP is the core module, this is a load-bearing ambiguity that must be resolved before the method can be reproduced or evaluated.
  2. [§3.2, Eqs. (5)-(11)] The architecture admits degenerate solutions in which Hα contains lexical content, so the interpretation of the BLEU gains as purification is not established. LSPK and LSNR in Eqs. (9)-(10) only push Hα to predict speaker identity and SNR; they never penalize Hα for also encoding content. LMI in Eq. (7) minimizes mutual information between Hγ and Hβ*, not between Hα and content, and it is weighted by λ2=0.01. If content leaks into Hα, the projection in Eq. (6) will remove it from Hγ, and minimizing LMI can then be satisfied by discarding content from Hγ—exactly the failure mode the method is intended to avoid. The paper's limitation section mentions coarse extraction granularity but not this possibility. A concrete test would be to train a content probe (e.g., an ASR or phone classifier) on Hα and show that it has near-chance accuracy, or to compare the projection against a random-subspace control; without such evidence, the causal story of 'purification' remains unsupported.
  3. [Tables 1-4] The paper claims statistical significance (p < 0.01) with asterisks, but it never describes the test, the number of independent runs, the variance, or how multiple directions were handled. This matters because the abstract's central claim is that SRPSE 'significantly improves' translation performance across all directions, and some differences in Table 1 are small (e.g., 26.9 vs. 27.0 for En-De in the multi-task setting). The authors should report a concrete significance procedure, such as paired bootstrap resampling over test sentences, together with standard deviations or confidence intervals; the same applies to the small differences in the ablation study in Table 5.
  4. [§1 and §3.2, Eqs. (9)-(10)] The claim that the method 'does not require transcriptions or additional annotations' is overstated. The transcript-free setting still uses LSPK in Eq. (9), which requires a speaker ID for every training utterance, and LSNR in Eq. (10), which requires the sampled perturbation labels. Speaker IDs are a form of metadata annotation that may not be available for unwritten or low-resource languages, weakening the stated motivation. The paper should either qualify this claim or provide results for the variant without speaker supervision (the ablation in Table 5 suggests this costs only about 0.2 BLEU on En-De, but it is not evaluated across directions).
minor comments (6)
  1. [Table 3 caption] The caption says 'expended setting'; this should be 'expanded setting'.
  2. [Appendix D, Algorithm 1] Algorithm 1 says 'Calculate the total loss (Equation 3)', but Equation 3 is the MT loss, not the total SRPSE loss; the intended reference is Eq. (12) or Eq. (13).
  3. [§5.2, Figure 7] The sentence 'The averaged G value across 5 subsets in Figure 1 is 4.05, while our model is 3.8' is ambiguous because G is defined per sentence in Eq. (1); the authors should specify whether they report the mean over all dev sentences or the mean of the five subset means.
  4. [§5.1, Figure 6] The t-SNE and cosine-similarity analysis is based on a single model run and provides no measure of variability; it should be treated as a qualitative illustration rather than quantitative evidence for purification.
  5. [Appendix E] The paper states that λ2 was not tuned but fixed to 0.01 following Yang et al. (2022). Given that LMI is one of the core objectives, a sensitivity analysis over λ2 would strengthen the method section.
  6. [General] The authors do not mention code or model release; for reproducibility, providing the implementation and trained model checkpoints would be valuable.

Circularity Check

1 steps flagged · score 2.0 of 10

Main BLEU claims are independent external evaluations; the only partly circular element is the Sec. 5.2 robustness check, which re-measures essentially the LCONSIS training objective.

  1. fitted input called prediction [Section 5.2, Eqs. (1) and (11)]
    "G =∥ Avg(fe(s)) − Avg(fe(˜s)) ∥2, (1) ... LCONSIS = |D|X∥ Avg(Hγ) − Avg( eHγ) ∥2 . (11) ... We conduct the same experiment as in Section 2, using voice conversion to perturb the speech input to assess the robustness of our model. ... The averaged G value across 5 subsets in Figure 1 is 4.05, while our model is 3.8."

    The robustness metric G (Eq. 1) is the average-pooled encoder representation distance between original and perturbed speech. LCONSIS (Eq. 11) directly minimizes the analogous Hγ-versus-eHγ distance for perturbed inputs during training, so reporting a reduced G and a smaller BLEU gap under perturbation in Sec. 5.2 is the expected consequence of optimizing that same objective, not an independent confirmation that 'purification' caused the gains. This is only partial, because G is measured on the textual-encoder output (fe) rather than directly on Hγ, and the test perturbation is voice conversion rather than the training-time noise/pitch/time-stretch perturbations. The central BLEU results in Tables 1-3 are held-out external evaluations and are not circular.

full rationale

The paper's central claim—BLEU improvement on MuST-C and CoVoST-2—is tested on held-out tst-COMMON/test sets against external baselines (Tables 1-4), so the main result is self-contained and not obtained by fitting test labels or by a self-citation chain. The OPP projection (Eqs. 5-6) is a fixed linear operation with no fitted constants, and the 'content-agnostic' supervision (speaker ID/SNR) is external to the test metric. The one partially circular element is the robustness analysis in Sec. 5.2, which measures essentially the same averaged representation distance that LCONSIS (Eq. 11) was trained to minimize; showing this distance shrinks is a sanity check on the optimization objective rather than an independent prediction. The paper's own limitations admit incomplete coverage of content-agnostic factors and coarse extraction granularity, but those are completeness caveats, not circular reductions. Overall circularity is therefore low.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method relies on a linear-separability assumption between content and style, on speaker and SNR labels as proxies for all content-agnostic factors, and on several hand-tuned hyperparameters. It introduces no new entities.

free parameters (5)
  • lambda1 (weight of consistency loss LCONSIS) = 1.0
    Chosen via hyperparameter selection on MuST-C En-De dev set (Appendix E, Figure 10).
  • lambda2 (weight of mutual information loss LMI) = 0.01
    Set following Yang et al. (2022), not tuned in this paper.
  • Nalpha (number of CA-Enc layers) = 1
    Selected from {1,2,3}; performance nearly flat, so minimal chosen (Appendix E, Figure 8).
  • Nbeta (number of CI-Enc layers) = 1
    Selected on MuST-C En-De; Nbeta=1 best (Appendix E, Figure 9).
  • Perturbation ranges (epsilon, mu, tau) = epsilon in {5,10,20,50,+inf}, mu in {-1,0,1}, tau in {0.8,0.9,1.0,1.1,1.2}
    Hand-selected design choices in Section 3.2, not swept.
assumptions (4)
  • domain assumption Speech representation is a sum of content-relevant and content-agnostic information, with content-agnostic information being removable.
    Introduced in Introduction and Section 2; this decomposition motivates the whole method.
  • domain assumption Speaker ID and SNR are adequate supervision signals for content-agnostic information.
    Used in Equations 9-10; the CA-Enc is trained to predict these labels.
  • ad hoc to paper Content and content-agnostic factors are linearly separable in the representation space so that orthogonal projection removes all agnostic information.
    Equations 5-6 assume this; no theoretical justification is given.
  • domain assumption Wav2vec2.0 features are suitable for ST and are comparable to baselines.
    Shared with baseline W2V2-Transformer; standard in the field.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Representation Purification for End-to-End Speech Translation." pith.science (2026). https://pith.science/paper/OKPIKTRI

@misc{pith2026241204266,
  author       = {Pith},
  title        = {Pith review of: Representation Purification for End-to-End Speech Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKPIKTRI}},
  note         = {Machine review of arXiv:2412.04266}
}
read the original abstract

Speech-to-text translation (ST) is a cross-modal task that involves converting spoken language into text in a different language. Previous research primarily focused on enhancing speech translation by facilitating knowledge transfer from machine translation, exploring various methods to bridge the gap between speech and text modalities. Despite substantial progress made, factors in speech that are not relevant to translation content, such as timbre and rhythm, often limit the efficiency of knowledge transfer. In this paper, we conceptualize speech representation as a combination of content-agnostic and content-relevant factors. We examine the impact of content-agnostic factors on translation performance through preliminary experiments and observe a significant performance deterioration when content-agnostic perturbations are introduced to speech signals. To address this issue, we propose a \textbf{S}peech \textbf{R}epresentation \textbf{P}urification with \textbf{S}upervision \textbf{E}nhancement (SRPSE) framework, which excludes the content-agnostic components within speech representations to mitigate their negative impact on ST. Experiments on MuST-C and CoVoST-2 datasets demonstrate that SRPSE significantly improves translation performance across all translation directions in three settings and achieves preeminent performance under a \textit{transcript-free} setting.

Figures

Figures reproduced from arXiv: 2412.04266 by the authors.

Figure 1
Figure 1. BLEU scores on MuST-C En-De dev subsets. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Averaged information entropy of cross￾attention weights. 2 Preliminary Analysis In this section, we examine the impact of content￾agnostic perturbations on the ST model. Typi￾cally, an ST dataset that contains triplet data can be formed as D = {(s, x, y)}, where s, x, y de￾note source speech, transcription, and translation, respectively. To perturb in the content-agnostic aspects of speech while preserving the conte… view at source ↗
Figure 4
Figure 4. Overview of our proposed framework. The text embedding and MT forward path are deprecated during inference or training in the transcript-free setting. in Section 3.2. An overview of our framework is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Diagram of OPP Module. It consists of two [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Bivariate KDE contour plot of speech and text [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: BLEU scores on MuST-C En-De dev subsets with our SRPSE. Perturbed and raw denote the BLEU scores are calculated with perturbed audio ˜s and raw audio s respectively. The Green bar denotes the G value. from these experimental findings, SRPSE achieves better performance …
Figure 8
Figure 8. Figure 8: BLEU scores with different number of CA [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: BLEU scores with different number of CI-Enc [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: BLEU scores with different λ1 on MuST-C En-De tst-COMMON set. Here the x-axis is the weight of LCONSIS. setup of λ1, λ2, Nα, and Nβ is 1.0, 0.01, 3, and 3, respectively. The results of selecting Nα are shown in Fig￾ure 8. We find the performance has almost no changes …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 38 canonical work pages

  1. [1]

    Ashkan Alinejad and Anoop Sarkar. 2020. Effectively pretraining a speech translation decoder with machine translation data. In Proc. of EMNLP, pages 8014--8020

    work page 2020

  2. [2]

    Ardila, M

    R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4211--4215

    2020

  3. [3]

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33:12449--12460

    2020

  4. [4]

    Content-Context Factorized Representations for Automated Speech Recognition

    David M. Chan and Shalini Ghosh. 2022. https://arxiv.org/abs/2205.09872 Content-context factorized representations for automated speech recognition . Preprint, arXiv:2205.09872

    work page Pith review arXiv 2022

  5. [5]

    Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. 2020. https://proceedings.mlr.press/v119/cheng20b.html CLUB : A contrastive log-ratio upper bound of mutual information . In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1779--1788. PMLR

    work page 2020

  6. [6]

    Xuxin Cheng, Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, and Yuexian Zou. 2023. M 3 st: Mix at three levels for speech translation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE

    2023

  7. [7]

    Ju-chieh Chou, Cheng-chieh Yeh, and Hung-yi Lee. 2019. One-shot voice conversion by separating speaker and content representations with instance normalization. arXiv preprint arXiv:1904.05742

    arXiv 2019

  8. [8]

    Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi

    Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. https://doi.org/10.18653/v1/N19-1202 M u ST - C : a M ultilingual S peech T ranslation C orpus . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and ...

    doi:10.18653/v1/n19-1202 2019

Show all 67 references
  1. [9]

    Qianqian Dong, Rong Ye, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. 2021. Listen, understand and translate: Triple supervision decouples end-to-end speech-to-text translation. In Proc. of AAAI, volume 35, pages 12749--12759

    2021

  2. [10]

    Yichao Du, Zhirui Zhang, Weizhi Wang, Boxing Chen, Jun Xie, and Tong Xu. 2022. Regularizing end-to-end speech translation with triangular decomposition agreement. In Proc. of AAAI, volume 36, pages 10590--10598

    2022

  3. [11]

    Qingkai Fang and Yang Feng. 2023. https://doi.org/10.18653/v1/2023.acl-long.251 Back translation for speech-to-text translation without transcripts . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4567-...

    2023 doi

  4. [12]

    Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. Stemm: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7050--7062

    2022

  5. [13]

    Chi Han, Mingxuan Wang, Heng Ji, and Lei Li. 2021. Learning shared semantic space for speech-to-text translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2214--2225

    2021

  6. [14]

    Chak Ho Chan, Kaizhi Qian, Yang Zhang, and Mark Hasegawa-Johnson. 2022. https://doi.org/10.1109/ICASSP43922.2022.9747763 Speechsplit2.0: Unsupervised speech disentanglement for voice conversion without tuning autoencoder bottlenecks . In ICASSP 2022 - 2022 IEEE International C...

    2022

  7. [15]

    Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, and Shinji Watanabe. 2020. Espnet-st: All-in-one speech translation toolkit. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstration...

    2020

  8. [16]

    Sathish Indurthi, Shamil Chollampatt, Ravi Agrawal, and Marco Turchi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.560 CLAD - ST : Contrastive learning with adversarial data for robust speech translation . In Proceedings of the 2023 Conference on Empirical Methods in Natu...

    2023 doi

  9. [17]

    Sathish Indurthi, Houjeung Han, Nikhil Kumar Lakumarapu, Beomseok Lee, Insoo Chung, Sangha Kim, and Chanwoo Kim. 2020. End-end speech-to-text translation with modality agnostic meta-learning. In Proc. of ICASSP, pages 7904--7908. IEEE

    2020

  10. [18]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2017. https://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . Preprint, arXiv:1412.6980

    2017 arXiv

  11. [19]

    Yeqiu Kong, Zhongwei Xu, and Meng Mei. 2023. https://doi.org/10.3390/s23167282 Cross-domain sentiment analysis based on feature projection and multi-source attention in iot . Sensors, 23(16)

    2023 doi

  12. [20]

    Taku Kudo and John Richardson. 2018. https://doi.org/10.18653/v1/D18-2012 S entence P iece: A simple and language independent subword tokenizer and detokenizer for neural text processing . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processin...

    2018 doi

  13. [21]

    Tsz Kin Lam, Shigehiko Schamoni, and Stefan Riezler. 2022. Sample, translate, recombine: Leveraging audio alignments for data augmentation in end-to-end speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Sh...

    2022

  14. [22]

    Phuong-Hang Le, Hongyu Gong, Changhan Wang, Juan Pino, Benjamin Lecouteux, and Didier Schwab. 2023. https://proceedings.mlr.press/v202/le23a.html Pre-training for speech translation: CTC meets optimal transport . In Proceedings of the 40th International Conference on Machine L...

    2023

  15. [23]

    Keon Lee, Kyumin Park, and Daeyoung Kim. 2021. https://arxiv.org/abs/2103.09474 Styler: Style factor modeling with rapidity and robustness via speech decomposition for expressive and controllable neural text to speech . Preprint, arXiv:2103.09474

    2021 arXiv

  16. [24]

    Yikun Lei, Zhengshan Xue, Xiaohu Zhao, Haoran Sun, Shaolin Zhu, Xiaodong Lin, and Deyi Xiong. 2023. https://doi.org/10.18653/v1/2023.findings-acl.195 CKDST : Comprehensively and effectively distill knowledge from machine translation to end-to-end speech translation . In Findin...

    2023 doi

  17. [25]

    Wenbiao Li, Ziyang Wang, and Yunfang Wu. 2023. https://arxiv.org/abs/2210.10305 A unified neural network model for readability assessment with feature projection and length-balanced loss . Preprint, arXiv:2210.10305

    2023 arXiv

  18. [26]

    Yuchen Liu, Hao Xiong, Jiajun Zhang, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019. End-to-end speech translation with knowledge distillation. Proc. Interspeech 2019, pages 1128--1132

    2019

  19. [27]

    Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2020. https://arxiv.org/abs/2010.14920 Bridging the modality gap for speech-to-text translation . Preprint, arXiv:2010.14920

    2020 arXiv

  20. [28]

    Chenggang Mi, Lei Xie, and Yanning Zhang. 2022. Improving data augmentation for low resource speech-to-text translation with diverse paraphrasing. Neural Networks, 148:194--205

    2022

  21. [29]

    Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations

    2019

  22. [30]

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. https://doi.org/10.1109/ICASSP.2015.7178964 Librispeech: An asr corpus based on public domain audio books . In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page...

    2015

  23. [31]

    Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D

    Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. https://doi.org/10.21437/Interspeech.2019-2680 SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition . In Proc. Interspeech 2019, pages 2613--2617

    2019 doi

  24. [32]

    Juan Pino, Qiantong Xu, Xutai Ma, Mohammad Javad Dousti, and Yun Tang. 2020. Self-training for end-to-end speech translation

    2020

  25. [33]

    Maja Popovi \'c . 2017. https://doi.org/10.18653/v1/W17-4770 chr F ++: words helping character n-grams . In Proceedings of the Second Conference on Machine Translation, pages 612--618, Copenhagen, Denmark. Association for Computational Linguistics

    2017 doi

  26. [34]

    Matt Post. 2018. https://www.aclweb.org/anthology/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Belgium, Brussels. Association for Computational Linguistics

    2018

  27. [35]

    Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, and David Cox. 2020. https://proceedings.mlr.press/v119/qian20a.html Unsupervised speech decomposition via triple information bottleneck . In Proceedings of the 37th International Conference on Machine Learning, volu...

    2020

  28. [36]

    Qi Qin, Wenpeng Hu, and Bing Liu. 2020. https://doi.org/10.18653/v1/2020.acl-main.726 Feature projection for improved text classification . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8161--8171, Online. Association for Com...

    2020 doi

  29. [37]

    Ricardo Rei, Jos \'e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.52 COMET -22: Unbabel- IST 2022 submission for the metrics shared task . In Pro...

    2022

  30. [38]

    RJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron Weiss, Rob Clark, and Rif A. Saurous. 2018. https://proceedings.mlr.press/v80/skerry-ryan18a.html Towards end-to-end prosody transfer for expressive speech synthesis with tacotron . In Proce...

    2018

  31. [39]

    Matthias Sperber, Graham Neubig, Jan Niehues, and Alex Waibel. 2017. Neural lattice-to-sequence models for uncertain inputs. In Proc. of EMNLP

    2017

  32. [40]

    Matthias Sperber, Graham Neubig, Ngoc-Quan Pham, and Alex Waibel. 2019. https://doi.org/10.18653/v1/P19-1115 Self-attentional models for lattice inputs . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1185--1197, Florence, Ita...

    2019 doi

  33. [41]

    Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. 2021. Improving speech translation by understanding and learning from the auxiliary text translation task. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th I...

    2021

  34. [42]

    Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11)

    2008

  35. [43]

    Hari Krishna Vydana, Martin Karafi \'a t, Katerina Zmolikova, Luk \'a s Burget, and Honza C ernock \`y . 2021. Jointly trained transformers models for spoken language translation. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICA...

    2021

  36. [44]

    Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020 a . Fairseq s2t: Fast speech-to-text modeling with fairseq. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th Internatio...

    2020

  37. [45]

    Changhan Wang, Anne Wu, and Juan Pino. 2020 b . https://arxiv.org/abs/2007.10310 Covost 2: A massively multilingual speech-to-text translation corpus . Preprint, arXiv:2007.10310

    2020 arXiv

  38. [46]

    Chengyi Wang, Yu Wu, Shujie Liu, Zhenglu Yang, and Ming Zhou. 2020 c . https://doi.org/10.1609/aaai.v34i05.6452 Bridging the gap between pre-training and fine-tuning for end-to-end speech translation . Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9161--9168

    2020 doi

  39. [47]

    Yan Xia, Hai Huang, Jieming Zhu, and Zhou Zhao. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/c89f09849eb5af489abb122394ff0f0b-Paper-Conference.pdf Achieving cross modal generalization with multimodal unified representation . In Advances in Neural Informatio...

    2023

  40. [48]

    Jiu-Cheng Xie, Chi-Man Pun, and Kin-Man Lam. 2022. https://doi.org/10.1109/TIFS.2022.3142998 Implicit and explicit feature purification for age-invariant facial representation learning . IEEE Transactions on Information Forensics and Security, 17:399--412

    2022

  41. [49]

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020. https://proceedings.mlr.press/v119/xiong20b.html On layer normalization in the transformer architecture . In Proceedings of the 37th Internatio...

    2020

  42. [50]

    Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju, Tong Xiao, and Jingbo Zhu. 2021. Stacked acoustic-and-textual encoding: Integrating the pre-trained models into speech translation encoders. In Proc. ACL, pages 2619--2630

    2021

  43. [52]

    SiCheng Yang, Methawee Tantrawenith, Haolin Zhuang, Zhiyong Wu, Aolan Sun, Jianzong Wang, Ning Cheng, Huaizhen Tang, Xintao Zhao, Jie Wang, and Helen Meng. 2022. https://doi.org/10.21437/Interspeech.2022-571 Speech Representation Disentanglement with Adversarial Mutual Informa...

    2022 doi

  44. [53]

    Yao-Yuan Yang, Moto Hira, Zhaoheng Ni, Anjali Chourdia, Artyom Astafurov, Caroline Chen, Ching-Feng Yeh, Christian Puhrsch, David Pollack, Dmitriy Genzel, Donny Greenberg, Edward Z. Yang, Jason Lian, Jay Mahadeokar, Jeff Hwang, Ji Chen, Peter Goldsborough, Prabhat Roy, Sean Na...

    2021 arXiv

  45. [54]

    Rong Ye, Mingxuan Wang, and Lei Li. 2021 a . https://doi.org/10.21437/Interspeech.2021-1065 End-to-End Speech Translation via Cross-Modal Progressive Training . In Proc. Interspeech 2021, pages 2267--2271

    2021 doi

  46. [55]

    Rong Ye, Mingxuan Wang, and Lei Li. 2021 b . End-to-end speech translation via cross-modal progressive training. arXiv preprint arXiv:2104.10380

    2021 arXiv

  47. [56]

    Rong Ye, Mingxuan Wang, and Lei Li. 2022. Cross-modal contrastive learning for speech translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5099--5113

    2022

  48. [57]

    Yuxuan Yuan, Yue Zhou, and Xiaodong Shi. 2024. Memory-augmented speech-to-text translation with multi-scale context translation strategy. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12727--12731. IEEE

    2024

  49. [58]

    Biao Zhang, Barry Haddow, and Rico Sennrich. 2022. https://proceedings.mlr.press/v162/zhang22i.html Revisiting end-to-end speech-to-text translation from scratch . In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Lea...

    2022

  50. [59]

    Dong Zhang, Rong Ye, Tom Ko, Mingxuan Wang, and Yaqian Zhou. 2023 a . Dub: Discrete unit back-translation for speech translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7147--7164

    2023

  51. [60]

    Hao Zhang, Nianwen Si, Yaqi Chen, Wenlin Zhang, Xukui Yang, Dan Qu, and Zhen Li. 2023 b . https://doi.org/10.1109/ICASSP49357.2023.10096899 Decoupled non-parametric knowledge distillation for end-to-end speech translation . In ICASSP 2023 - 2023 IEEE International Conference o...

    2023

  52. [61]

    Yuhao Zhang, Kaiqi Kou, Bei Li, Chen Xu, Chunliang Zhang, Tong Xiao, and Jingbo Zhu. 2024. https://doi.org/10.1109/ICASSP48485.2024.10447494 Soft alignment of modality space for end-to-end speech translation . In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, S...

    2024

  53. [62]

    Yuhao Zhang, Chen Xu, Bei Li, Hao Chen, Tong Xiao, Chunliang Zhang, and Jingbo Zhu. 2023 c . Rethinking and improving multi-task learning for end-to-end speech translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10753--10765

    2023

  54. [63]

    Xiaohu Zhao, Haoran Sun, Yikun Lei, Shaolin Zhu, and Deyi Xiong. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.394 CCSRD : Content-centric speech representation disentanglement learning for end-to-end speech translation . In Findings of the Association for Computationa...

    2023 doi

  55. [64]

    Yan Zhou, Qingkai Fang, and Yang Feng. 2023. Cmot: Cross-modal mixup via optimal transport for speech translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7873--7887

    2023

  56. [65]

    Yue Zhou, Yuxuan Yuan, and Xiaodong Shi. 2024. A multitask co-training framework for improving speech translation by leveraging speech recognition and machine translation tasks. Neural Computing and Applications, pages 1--16

    2024

  57. [66]

    Ziyue Zhu, Zhao Zhang, Zheng Lin, Xing Sun, and Ming-Ming Cheng. 2023. https://doi.org/10.1109/TPAMI.2023.3234586 Co-salient object detection with co-representation purification . IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8193--8205

    2023

  58. [67]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  59. [68]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.