Pith. sign in

REVIEW 4 cited by

Direct speech-to-speech translation with discrete units

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2107.05604 v2 pith:COVXGQ6I submitted 2021-07-12 cs.CL cs.LGeess.AS

Ann Lee , Peng-Jen Chen , Changhan Wang , Jiatao Gu , Sravya Popuri , Xutai Ma , Adam Polyak , Yossi Adi
show 4 more authors
Qing He Yun Tang Juan Pino Wei-Ning Hsu
This is my paper · ORCID
classification cs.CLcs.LGeess.AS
keywords speechtextmodeltranslationdirectdiscretes2sttarget
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation. We tackle the problem by first applying a self-supervised discrete speech encoder on the target speech and then training a sequence-to-sequence speech-to-unit translation (S2UT) model to predict the discrete representations of the target speech. When target text transcripts are available, we design a joint speech and text training framework that enables the model to generate dual modality output (speech and text) simultaneously in the same inference pass. Experiments on the Fisher Spanish-English dataset show that the proposed framework yields improvement of 6.7 BLEU compared with a baseline direct S2ST model that predicts spectrogram features. When trained without any text transcripts, our model performance is comparable to models that predict spectrograms and are trained with text supervision, showing the potential of our system for translation between unwritten languages. Audio samples are available at https://facebookresearch.github.io/speech_translation/direct_s2st_units/index.html .

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation System

    eess.AS 2026-07 conditional novelty 5.0 of 10

    An open, modular cascaded system (streaming ASR + MT + prompt-conditioned TTS) preserves speaker identity in long-form multi-speaker translation, at higher latency and slightly lower translation quality than proprietary APIs.

  2. When End-to-End is Overkill: Rethinking Cascaded Speech-to-Text Translation

    cs.CL 2025-02 conditional novelty 5.0 of 10

    A cascaded speech-to-text translation model that feeds five aligned ASR candidates and self-supervised speech units to a translation model matches end-to-end performance on GigaST, with an English-to-Chinese BLEU of 38.1.

  3. Improving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translation

    cs.SD 2024-12 conditional novelty 5.0 of 10

    Adding a SyncNet-based lip-synchrony loss plus a duration loss to a pre-trained audio-visual speech-to-speech model improves lip-sync of overlaid translated audio on original videos across four language pairs.

  4. Speech to Speech Translation with Translatotron: A State of the Art Review

    cs.CL 2025-02 reject

    A survey of Translatotron speech-to-speech translation models that asserts, without evidence, that Translatotron 3 is the best choice for low-resource African languages.

Pith tools