Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Improving Language and Modality Transfer in Translation by Character-level Modeling

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Character-level modeling in a shared embedding space improves translation across languages and turns ASR audio into speech translation without parallel speech-translation data.

desk verdict The text-transfer half is a solid, controlled result; the speech 'state-of-the-art' claim sits on a 0.002 COMET margin and needs significance evidence before it can carry the abstract. read the letter →

arxiv 2505.24561 v1 pith:NNWTB4MG submitted 2025-05-30 cs.CL

Ioannis Tsiamas , David Dale , Marta R. Costa-jussà This is my paper
classification cs.CL
keywords character-levelmachinetranslationSONARembeddingspacezero-shotspeechcross-lingualtransfercross-modallow-resourcelanguagesCTCadapterMMSASR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes shifting the input side of a multilingual translation system from subword tokens to characters, inside SONAR's fixed-size sentence-embedding space. It claims that a character-level encoder trained to imitate SONAR teacher embeddings, with an objective that interpolates between source and target embeddings, transfers better across languages than comparable subword-based encoders, especially for low-resource and never-seen languages. The same character space lines up with the output vocabulary of MMS, a CTC speech recognizer covering 1,000+ languages, so a tiny adapter can connect the two with zero or minimal training. On FLEURS, this zero-shot speech translation surpasses supervised end-to-end and cascade baselines across 33 languages. If correct, translation coverage could expand to the long tail of languages using only text and ASR data, without parallel speech-translation data.

What carries the argument

The load-bearing mechanism is the pair of overlapping character vocabularies. charSONAR keeps only the single-character tokens of SONAR's 256K vocabulary as its input vocabulary $V_c$, and MMS's per-language CTC vocabularies satisfy $B^{(i)} \subset V_c$. Training uses a teacher-student MSE objective where the student maps a source sentence to the interpolated teacher embedding $(e_x + e_y)/2$ for a parallel pair $(x,y)$; together with family-token augmentation, this is what carries cross-lingual transfer. For speech, CTC-based compression turns MMS encoder outputs into character-like acoustic tokens, and the pretrained cross-modal adapter $E_{pt} = \mathrm{softmax}(A W^{(i)}) \mathrm{Emb}^{(i)}$ projects them into the charSONAR embedding space using MMS's own CTC classification layer $W^{(i)}$ and charSONAR's embedding table $\mathrm{Emb}^{(i)}$, so no new parameters are needed for the zero-shot baseline. A dual adapter adds a randomly initialized branch and learns a per-position weight between the two branches, which becomes useful when more ASR data is available.

What would settle it

Measure the coverage fraction $|B^{(i)} \cap V_c| / |B^{(i)}|$ for each of the 33 FLEURS languages, and feed audio for a language whose orthography uses a character absent from $V_c$ through the pretrained adapter; if COMET does not drop when such characters are common, the subset assumption is not load-bearing. Alternatively, retrain the subword encoder with the same interpolation objective and family tokens: if the char/subword gap on low-resource languages disappears, the paper's attribution of transfer to character-level modeling is wrong.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that character-level modeling is not merely a robustness trick but a transfer mechanism: a character-level encoder (charSONAR) trained against fixed-size SONAR embeddings outperforms an equally sized subword encoder across 75 languages on FLORES+, with the largest margin in the 21 lowest-resource languages, and generalizes better to 12 languages never seen in training. The paper attributes this to the interpolation objective, which maps source sentences to the midpoint of the source and target teacher embeddings, a region the authors find better for both translation and retrieval than either endpoint. The paper further discovers that the output space of a frozen multilingual CTC recognizer (MMS) is natively compatible with the character vocabulary of charSONAR, so a minimal pretrained adapter—softmax of CTC logits multiplied by the charSONAR embedding table—yields speech-to-text translation out of the box, outperforming Whisper, and, when trained on ASR data alone, surpassing the previous supervised state of the art SeamlessM4T on FLEURS for 33 languages.

Load-bearing premise

The pretrained speech adapter works only if every character in a language's MMS vocabulary already appears in charSONAR's character vocabulary; the paper states this subset condition but provides no coverage counts, so for any language whose writing system uses characters outside that set the out-of-the-box result silently loses tokens.

Editorial extensions

If this is right

  • The interpolation objective justifies training charSONAR on parallel data, mapping each source to the midpoint between source and target embeddings, which proves better than either endpoint for both translation and retrieval.
  • Speech translation for a new language can be enabled with audio-transcription pairs only; the adapter with ~0.2M parameters trained on Common Voice reaches or exceeds supervised baselines, and the zero-training pretrained adapter already beats Whisper on FLEURS.
  • For languages with very little ASR data (e.g., Occitan with ~0.3K examples), the random adapter collapses while the pretrained adapter retains most quality, showing that initialization carries the transfer in extreme low-resource conditions.
  • Extra data (2K FLEURS examples) and the dual adapter with a trainable weight between pretrained and random branches provide a scaling path to further gains across all resource groups.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the subset condition $B^{(i)} \subset V_c$ were replaced by an explicit character-coverage check or a character-to-character mapping, the pretrained adapter could be extended to any of MMS's 1,000+ languages, including those with scripts unseen in SONAR.
  • The interpolation result suggests that SONAR's embedding space has a 'translation sweet spot' between languages; a symmetric target-side interpolation or interpolation through the decoder could extend the benefit to target generation, though decoding at character level may be inefficient.
  • A testable prediction of the transfer-via-characters hypothesis: the charSONAR advantage over the subword model should grow monotonically with orthographic distance between source and target (script mismatch, morphological complexity); the paper's per-script tables hint at this but do not test it directly.
  • The paper's single-language control (English-only training) shows no char advantage, implying the gains are purely transfer; this predicts that transfer gains should vanish in a monolingual high-resource regime and reappear when the training set is skewed toward low-resource languages, a pattern worth verifying on a script-diverse held-out set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes charSONAR, a character-level text encoder trained by teacher-student distillation into SONAR's fixed-size embedding space, and a lightweight cross-modal adapter that connects MMS's CTC output to charSONAR, enabling zero-shot speech-to-text translation from ASR data alone. The text experiments on 75 languages (FLORES+) report consistent but small COMET gains over a subword-based SONAR baseline, especially for low-resource and unseen languages. The speech experiments on 33 languages (FLEURS) report COMET scores that match or slightly exceed SeamlessM4T-Large-v2, with variants trained on Common Voice and optionally FLEURS train. The paper also includes ablations of training objectives, augmentation, adapter variants, and efficiency analysis.

Significance. If the central claims hold, the paper makes a useful contribution: it shows that character-level encoding in a fixed-size semantic space can improve cross-lingual transfer for low-resource MT and enables a simple, data-efficient adapter for zero-shot speech translation. The controlled comparison against a fine-tuned subword SONAR baseline is a fair design, and the efficiency analysis in §5.5 is a concrete strength. However, the headline speech 'state-of-the-art' claim rests on a 0.002 COMET margin with no significance test, and the pretrained adapter relies on a vocab-subset assumption that is asserted but never verified. The text-transfer results are more solid, but their magnitude (0.004–0.013 COMET) needs uncertainty quantification.

major comments (4)
  1. [§6.2, Table 7] The abstract's claim of 'state-of-the-art results in speech-to-text translation on the FLEURS benchmark' is not supported by the evidence. Row 12 (PRETR, CV-only) reaches 0.875 COMET versus 0.873 for SeamlessM4T-Large-v2, a margin of 0.002. xCOMET-XL is a learned metric, and no confidence intervals, bootstrap, or per-segment significance test is reported, so a 0.002 difference is within noise. Furthermore, rows 14-15 add FLEURS train (speech-translation pairs), which contradicts the abstract's characterization of 'minimal supervision from ASR data' being sufficient for the SOTA result. The authors should either report significance evidence for the CV-only comparison or qualify the SOTA claim to the setting in which it is actually supported.
  2. [§3.3.2] The pretrained adapter assumes 'since B(i) is a subset of Vc' for every MMS language used. This is load-bearing because row 11 of Table 7 claims an out-of-the-box, zero-training speech translation result. No coverage statistics are provided: the paper does not verify for the 33 tested languages, let alone the 1,000+ MMS languages, that all CTC characters appear in charSONAR's 8K-character vocabulary. If any MMS language uses a character outside Vc (e.g., a diacritic or script variant), the pretrained adapter cannot represent it, and the zero-shot result loses those tokens. The paper should report per-language vocabulary coverage and either restrict the claim to verified languages or add a fallback for unseen characters.
  3. [Table 7, row 13] The dual adapter (DUAL) trained on Common Voice collapses on low-resource languages: row 13 reports 0.615 COMET for the low group, versus 0.837 for the pretrained adapter in row 12 and 0.833 for the cascade baseline in row 9. This is in tension with §6.1, which concludes that 'our proposed dual adapter surpasses them both' based on four languages. The paper does not discuss this failure mode, and the abstract's blanket claim of superior speech adaptation is therefore not accurate for the DUAL variant under the CV-only condition. The authors should either restrict the DUAL claim to medium/high-resource settings or analyze why the randomly-initialized branch causes a catastrophic drop in low-resource languages.
  4. [Table 3] The central text claim—that charSONAR 'achieves better language transfer than traditional subword-based models'—rests on COMET differences of 0.004–0.013 and xSIM++ differences of 0.3–0.7. The comparisons against SONAR-75 are controlled (same data, same training budget), which is good, but no significance testing is reported. Given the small magnitude of the differences and the use of a learned metric, the authors should provide paired bootstrap confidence intervals or per-language significance tests, at least for the headline Low and New language groups.
minor comments (5)
  1. [Abstract] The phrase 'albeit being a zero-shot model with minimal supervision from ASR data' is ambiguous: rows 14-15 use FLEURS train, which includes speech-translation supervision, so 'zero-shot' needs to be defined per row.
  2. [§3.2, Eq. (1)] The interpolation objective averages teacher embeddings ex and ey, but the paper does not define how the teacher embeddings are obtained for target sentences without parallel data in the monolingual case; this is only relevant for the 'Interpolation' variant, but a clarifying sentence would help.
  3. [§5.1, Table 1] The row labels 'recon', 'trans', and 'interpol' are not expanded in the table caption; adding a brief definition would improve readability.
  4. [§6.2, Table 7] Row 11 (PRETR, 0 train params) and row 12 are both labeled 'Speech-charSONAR-75 - PRETR'; the difference is the training data. Distinguishing the two with row names such as 'PRETR (untrained)' and 'PRETR (trained on CV)' would prevent confusion.
  5. [Appendix, Table 10] For some languages, the MT data column reports '0.1' (e.g., arg_Latn) and ASR data '0.0'; the units vary (M vs. K), and a consistent notation such as '0.1M' and '0.0K' would be clearer.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: charSONAR and the speech adapter are trained toward SONAR teacher embeddings, but the central comparisons are against equally trained subword baselines and external supervised systems, so the results are not forced by construction.

full rationale

The derivation chain is not circular. charSONAR is trained to minimize MSE against SONAR teacher embeddings, including the interpolation objective L_interpol = MSE(c_x, (e_x + e_y) / 2), so the student does inherit properties of the teacher space. However, the paper's central text claim is a controlled comparison against a subword model trained with the same setup: "We also train a comparable subword-based model by further fine-tuning SONAR on the 75 languages with the same setup as we did for charSONAR." Because SONAR-75 receives the same interpolation objective, the observed delta in Table 3 (0.889 vs 0.882 for low-resource, 0.917 vs 0.913 overall) isolates the character-level input representation from the teacher-target choice, rather than reducing to the teacher by construction. The speech adapter is a designed initialization E_pt = softmax(A W(i)) Emb(i) exploiting the subset relation B(i) ⊂ Vc; the zero-training row is an empirical measurement, not a definitional identity, and the headline speech result (Table 7, row 12: 0.875 COMET vs 0.873 for SeamlessM4T-Large-v2) is trained only on Common Voice ASR data and compared against externally trained supervised systems. The only overlapping-author citation, ZeroSwot (Tsiamas et al., 2024), is cited as prior paradigm ("we also follow the paradigm of ZeroSwot"), but the optimization here uses a different MSE objective, and no load-bearing claim is justified solely by that citation. The 0.002 COMET margin and the lack of significance testing are statistical-evidence concerns, not circularity, and under the review rules they do not raise the circularity score.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

Most of the load is carried by the SONAR teacher space and MMS acoustic encoder, both from the authors' own institution and cited without independent verification. The hand-chosen hyperparameters (noise probabilities, BLASER threshold, interpolation coefficient, family-token rate) are reasonable but not derived. The key structural assumption, B(i) subset of Vc, is asserted in Section 3.3.2 without coverage tests.

free parameters (6)
  • BLASER 2.0 filtering threshold = 4.0 main, 4.5 exploration
    Parallel data pairs with BLASER score below threshold are discarded; the choice affects training data quality and volume (Figure 3).
  • ASR-like noise probabilities = pnorm=0.25, pnoise=0.125, per-char rate 0.0025
    Tuned on a small Common Voice validation set to approximate MMS character error rate, affecting charSONAR robustness.
  • Interpolation coefficient = 0.5
    The target teacher embedding is the unweighted average (ex+ey)/2 in Eq. 1, chosen without search, and it determines the training target.
  • Family token replacement probability = 0.2
    During training, language token is replaced with subgroup token with 20% probability to enable zero-shot encoding of new languages.
  • Upsampling temperature = 0.5
    Language upsampling temperature controls how much low-resource languages are represented in the batch (Section 4.3).
  • Dual adapter hidden dimension and MLP inner dim = dh=1024, inner=64
    Capacity of the randomly initialized adapter branch, chosen to balance data efficiency and capacity (Section 3.3.2).
assumptions (5)
  • domain assumption SONAR's fixed-size embedding space is a reliable teacher for sentence-level distillation across languages, such that matching its embeddings yields good translation behavior when decoded by the SONAR decoder.
    The entire method is a teacher-student distillation into SONAR; if SONAR embeddings were not a good semantic target, the student would not translate well. Invoked in Sections 3.2 and 3.3.
  • ad hoc to paper The MMS CTC output vocabulary B(i) is a subset of the charSONAR character vocabulary Vc for every language used in the pretrained adapter.
    Section 3.3.2 states 'since B(i) is a subset of Vc, we can connect the two spaces'. The zero-training adapter depends on this; no coverage statistics are given for the 33 tested or 1,000+ claimed languages.
  • domain assumption CTC frame-wise argmax labels and the subsequent averaging compression preserve the linguistic content needed for translation.
    Section 3.3.1 uses CTC-based compression to shorten the acoustic representation; if the argmax collapses pronunciation variants, translation quality suffers.
  • domain assumption The SONAR decoder, trained for text inputs, can decode from speech-derived embeddings that are pushed into the same space.
    No finetuning of the decoder is performed for speech; the method assumes the frozen decoder handles the in-domain embeddings. Invoked in Section 3.3.2 and at inference.
  • domain assumption Language family subgrouping is a valid prior for zero-shot encoding of unseen languages.
    The use of subgroup tokens for unseen languages assumes that the family token captures the linguistic properties needed for embedding (Section 5.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Language and Modality Transfer in Translation by Character-level Modeling." pith.science (2026). https://pith.science/paper/NNWTB4MG

@misc{pith2026250524561,
  author       = {Pith},
  title        = {Pith review of: Improving Language and Modality Transfer in Translation by Character-level Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NNWTB4MG}},
  note         = {Machine review of arXiv:2505.24561}
}
read the original abstract

Current translation systems, despite being highly multilingual, cover only 5% of the world's languages. Expanding language coverage to the long-tail of low-resource languages requires data-efficient methods that rely on cross-lingual and cross-modal knowledge transfer. To this end, we propose a character-based approach to improve adaptability to new languages and modalities. Our method leverages SONAR, a multilingual fixed-size embedding space with different modules for encoding and decoding. We use a teacher-student approach with parallel translation data to obtain a character-level encoder. Then, using ASR data, we train a lightweight adapter to connect a massively multilingual CTC ASR model (MMS), to the character-level encoder, potentially enabling speech translation from 1,000+ languages. Experimental results in text translation for 75 languages on FLORES+ demonstrate that our character-based approach can achieve better language transfer than traditional subword-based models, especially outperforming them in low-resource settings, and demonstrating better zero-shot generalizability to unseen languages. Our speech adaptation, maximizing knowledge transfer from the text modality, achieves state-of-the-art results in speech-to-text translation on the FLEURS benchmark on 33 languages, surpassing previous supervised and cascade models, albeit being a zero-shot model with minimal supervision from ASR data.

Figures

Figures reproduced from arXiv: 2505.24561 by the authors.

Figure 1
Figure 1. Approach for character-level and speech adap [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Cross-modal Adapters To the output of the cross-modal adapter E we prepend the corresponding language token em￾bedding, and append the embedding for end-of￾sentence from the charSONAR embedding table. After adding positional encoding, E is passed through the transformer layers of the (frozen) char￾SONAR encoder to obtain a speech embedding c z ∈ R dt . To train the adapter we use audio￾transcription pairs, and minim… view at source ↗
Figure 3
Figure 3. COMET scores vs. BLASER 2.0 filtering threshold for charSONAR and SONAR in FLORES dev. Results with the Ural/Cyrl group of 15 languages. COMET scores are average of X→Eng for all the 15 languages in the group. in all categories, apart from the single group that contains the Greek language, and only in cross￾lingual similarity search. For the new languages, we notice that charSONAR performs better in all categories f… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLM-BT-Terms: Back-Translation as a Framework for Terminology Standardization and Dynamic Semantic Embedding

    cs.CL 2025-06 reject novelty 5.0 of 10

    LLM-based back-translation can validate and recommend standardized multilingual terminology with over 90 percent reported consistency in small case studies.

Reference graph

Works this paper leans on

51 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. https://aclanthology.org/2020.lrec-1.520/ Common Voice: A Massively-Multilingual Speech Corpus . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4218--4222, Marse...

    work page 2020

  2. [2]

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. https://proceedings.neurips.cc/paper/2020/file/92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdf wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations . In Advances in Neural Information Processing Systems, volume 33, pages 12449--12460. Curran Associates, Inc

    work page 2020

  3. [3]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. http://arxiv.org/abs/1409.0473 Neural Machine Translation by Jointly Learning to Align and Translate . In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

    arXiv 2015

  4. [4]

    Kris Cao. 2023. https://doi.org/10.18653/v1/2023.acl-long.326 What is the best recipe for character-level encoder-only modelling? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5924--5938, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.326 2023

  5. [5]

    Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen

    Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.236 When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4074--4096, Miami, Florida, USA. Association for C...

    doi:10.18653/v1/2024.emnlp-main.236 2024

  6. [6]

    Mingda Chen, Kevin Heffernan, Onur C elebi, Alexandre Mourachko, and Holger Schwenk. 2023. https://doi.org/10.18653/v1/2023.acl-short.10 xSIM++: An Improved Proxy to Bitext Mining Performance for Low-Resource Languages . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 101--109, Tor...

    work page doi:10.18653/v1/2023.acl-short.10 2023

  7. [7]

    Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. 2018. https://doi.org/10.18653/v1/D18-1461 Revisiting Character-Based Neural Machine Translation with Capacity and Compression . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4295--4305, Brussels, Belgium. Association for Comput...

    work page doi:10.18653/v1/d18-1461 2018

  8. [8]

    Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. https://doi.org/10.18653/v1/P16-1160 A Character-level Decoder without Explicit Segmentation for Neural Machine Translation . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1693--1703, Berlin, Germany. Association for Computat...

    work page doi:10.18653/v1/p16-1160 2016

Show all 51 references
  1. [9]

    Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021. https://doi.org/10.1109/ASRU51503.2021.9688253 w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training . In 2021 IEEE Automati...

    2021

  2. [10]

    Clark, Dan Garrette, Iulia Turc, and John Wieting

    Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. https://doi.org/10.1162/tacl_a_00448 Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation . Transactions of the Association for Computational Linguistics, 10:73--91

    2022 doi

  3. [11]

    Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. https://arxiv.org/abs/2205.12446 FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech . arXiv preprint arXiv:2205.12446

    2022 arXiv

  4. [12]

    Costa-juss \`a and Jos \'e A

    Marta R. Costa-juss \`a and Jos \'e A. R. Fonollosa. 2016. https://doi.org/10.18653/v1/P16-2058 Character-based Neural Machine Translation . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 357--361, Ber...

    2016 doi

  5. [13]

    Costa-juss \`a

    David Dale and Marta R. Costa-juss \`a . 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.943 BLASER 2.0: A Metric for Evaluation and Quality Estimation of Massively Multilingual Speech and Text Translation . In Findings of the Association for Computational Linguistics: E...

    2024 doi

  6. [14]

    Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023. https://arxiv.org/abs/2308.11466 SONAR: Sentence-Level Multimodal and Language-Agnostic Representations . Preprint, arXiv:2308.11466

    2023 arXiv

  7. [15]

    Lukas Edman, Gabriele Sarti, Antonio Toral, Gertjan van Noord, and Arianna Bisazza. 2024. https://doi.org/10.1162/tacl_a_00651 Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation . Transactions of the Association for Computational Li...

    2024 doi

  8. [16]

    Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. https://doi.org/10.18653/v1/2022.acl-long.486 STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguis...

    2022 doi

  9. [17]

    Marco Gaido, Mauro Cettolo, Matteo Negri, and Marco Turchi. 2021. https://doi.org/10.18653/v1/2021.eacl-main.57 CTC-based Compression for Direct Speech Translation . In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics:...

    2021 doi

  10. [18]

    Alex Graves, Santiago Fern\' a ndez, Faustino Gomez, and J\" u rgen Schmidhuber. 2006. https://doi.org/10.1145/1143844.1143891 Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks . In Proceedings of the 23rd International C...

    2006

  11. [19]

    Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andr \'e F

    Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andr \'e F. T. Martins. 2024. https://doi.org/10.1162/tacl_a_00683 xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection . Transactions of the Association for Co...

    2024 doi

  12. [20]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. https://proceedings.mlr.press/v97/houlsby19a.html Parameter-Efficient Transfer Learning for NLP . In Proceedings of the 36th In...

    2019

  13. [21]

    Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean

    Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. https://doi.org/10.1162/tacl_a_00065 Google`s Multilingual Neural Machine Translation...

    2017 doi

  14. [22]

    Sneha Kudugunta, Isaac Rayburn Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. https://openreview.net/forum?id=Y45ZCxslFx MADLAD-400: A Multilingual And Document-Level Large Audited Dataset . In Thirty-seventh ...

    2023

  15. [23]

    Phuong-Hang Le, Hongyu Gong, Changhan Wang, Juan Pino, Benjamin Lecouteux, and Didier Schwab. 2023. https://dl.acm.org/doi/10.5555/3618408.3619179 Pre-training for Speech Translation: CTC Meets Optimal Transport . In Proceedings of the 40th International Conference on Machine ...

    2023

  16. [24]

    Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. https://doi.org/10.1162/tacl_a_00067 Fully Character-Level Neural Machine Translation without Explicit Segmentation . Transactions of the Association for Computational Linguistics, 5:365--378

    2017 doi

  17. [25]

    Jind r ich Libovick \'y , Helmut Schmid, and Alexander Fraser. 2022. https://doi.org/10.18653/v1/2022.findings-acl.194 Why don`t people use character-level machine translation? In Findings of the Association for Computational Linguistics: ACL 2022, pages 2470--2485, Dublin, Ir...

    2022 doi

  18. [26]

    Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. 2015. https://arxiv.org/abs/1511.04586 Character-based Neural Machine Translation . Preprint, arXiv:1511.04586

    2015 arXiv

  19. [27]

    Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2020. https://arxiv.org/abs/2010.14920 Bridging the Modality Gap for Speech-to-Text Translation . Preprint, arXiv:2010.14920

    2020 arXiv

  20. [28]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled Weight Decay Regularization . In International Conference on Learning Representations

    2019

  21. [29]

    NLLB . 2024. https://doi.org/10.1038/s41586-024-07335-x Scaling Neural Machine Translation to 200 Languages . Nature, 630:841–846

    2024 doi

  22. [30]

    Siqi Ouyang, Rong Ye, and Lei Li. 2023. https://doi.org/10.18653/v1/2023.acl-long.216 WACO: Word-Aligned Contrastive Learning for Speech Translation . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3891...

    2023 doi

  23. [31]

    Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer. 2024. https://arxiv.org/abs/2412.09871 Byte Latent Transformer: Patch...

    2024 arXiv

  24. [32]

    Gabriel Peyré and Marco Cuturi. 2019. https://doi.org/10.1561/2200000073 Computational Optimal Transport: With Applications to Data Science . Foundations and Trends® in Machine Learning, 11(5-6):355--607

    2019 doi

  25. [33]

    Maja Popovi \'c . 2017. https://doi.org/10.18653/v1/W17-4770 chrF++: Words Helping Character N-grams . In Proceedings of the Second Conference on Machine Translation, pages 612--618, Copenhagen, Denmark. Association for Computational Linguistics

    2017 doi

  26. [34]

    Matt Post. 2018. https://www.aclweb.org/anthology/W18-6319 A Call for Clarity in Reporting BLEU Scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Belgium, Brussels. Association for Computational Linguistics

    2018

  27. [35]

    Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2023. https://arxiv.org/abs/2305.13516 Sca...

    2023 arXiv

  28. [36]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. https://arxiv.org/abs/2212.04356 Robust Speech Recognition via Large-Scale Weak Supervision . Preprint, arXiv:2212.04356

    2022 arXiv

  29. [37]

    SEAMLESS. 2025. https://doi.org/10.1038/s41586-024-08359-z Joint Speech and Text Machine Translation for up to 100 Languages . Nature, 637:587–593

    2025 doi

  30. [38]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural Machine Translation of Rare Words with Subword Units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  31. [39]

    Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. https://papers.nips.cc/paper_files/paper/2014/hash/5a18e133cbf9f257297f410bb7eca942-Abstract.html Sequence to Sequence Learning with Neural Networks . In Proceedings of the 28th International Conference on Neural Information...

    2014

  32. [40]

    Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. 2021 a . https://doi.org/10.18653/v1/2021.acl-long.328 Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task . In Proceedings of the 59th Annual Meeting of the Associ...

    2021 doi

  33. [41]

    Yun Tang, Juan Pino, Changhan Wang, Xutai Ma, and Dmitriy Genzel. 2021 b . https://doi.org/10.1109/ICASSP39728.2021.9415058 A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks . In ICASSP 2021 - 2021 IEEE International Conference on Acoustics...

    2021

  34. [42]

    Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler

    Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2022. https://openreview.net/forum?id=JtBRnrlOEFN Charformer: Fast Character Transformers via Gradient-based Subword Tokenization . In Inter...

    2022

  35. [43]

    G \'a llego, Jos \'e A

    Ioannis Tsiamas, Gerard I. G \'a llego, Jos \'e A. R. Fonollosa, and Marta R. Costa-juss \`a . 2024. https://doi.org/10.18653/v1/2024.findings-acl.847 Pushing the Limits of Zero-shot End-to-End Speech Translation . In Findings of the Association for Computational Linguistics: ...

    2024 doi

  36. [44]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is All you Need . In Advances in Ne...

    2017

  37. [45]

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. https://doi.org/10.1162/tacl_a_00461 ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models . Transactions of the Association for Computatio...

    2022 doi

  38. [46]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer . In Proceedings of the 2021 Conference ...

    2021 doi

  39. [47]

    Jichen Yang, Kai Fan, Minpeng Liao, Boxing Chen, and Zhongqiang Huang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.871 Towards Zero-shot Learning for End-to-end Cross-modal Translation Models . In Findings of the Association for Computational Linguistics: EMNLP 2023,...

    2023 doi

  40. [48]

    Rong Ye, Mingxuan Wang, and Lei Li. 2021. https://doi.org/10.21437/Interspeech.2021-1065 End-to-End Speech Translation via Cross-Modal Progressive Training . In Interspeech 2021, pages 2267--2271

    2021 doi

  41. [49]

    Rong Ye, Mingxuan Wang, and Lei Li. 2022. https://doi.org/10.18653/v1/2022.naacl-main.376 Cross-modal Contrastive Learning for Speech Translation . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

    2022 doi

  42. [50]

    Yuhao Zhang, Chen Xu, Bei Li, Hao Chen, Tong Xiao, Chunliang Zhang, and Jingbo Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.663 Rethinking and Improving Multi-task Learning for End-to-end Speech Translation . In Proceedings of the 2023 Conference on Empirical Methods...

    2023 doi

  43. [51]

    Jinming Zhao, Vineel Pratap, and Michael Auli. 2024. https://arxiv.org/abs/2407.17852 Scaling A Simple Approach to Zero-Shot Speech Recognition . Preprint, arXiv:2407.17852

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.