Pith. sign in

REVIEW 2 major objections 4 minor 300 references

Streaming speech-to-speech translation is trainable from ~2,000 hours of paired audio when a joint text-code trajectory schedule and a two-stream Thinker–Talker design replace separate speech-side controllers.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:37 UTC pith:4EMQOHVS

load-bearing objection Solid empirical recipe for data-efficient streaming S2ST; the main qualification is a missing quantification of alignment-error residue in the trajectories, not a demonstrated flaw. the 2 major comments →

arxiv 2607.19810 v1 pith:4EMQOHVS submitted 2026-07-22 cs.SD

SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision

Rongshen He , Xinyu Liang , Dekun Chen , Jiaqi Li , Mingjie Chen , Zhizheng Wu This is my paper
classification cs.SD
keywords streaming speech-to-speech translationtrajectory supervisionjoint text-code commitmentThinker–Talker factorizationsemantic codesdata efficiencysimultaneous interpretationlatency–quality trade-off
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the main obstacle to streaming speech-to-speech translation (S2ST) is not reasoning capacity but supervision structure. It proposes a training recipe in which a speech language model is taught to translate continuously using only about 2,000 hours of paired cross-lingual audio, layered on much more abundant ASR, speech-to-text, machine-translation, and TTS data. The central device is joint text-code trajectory supervision: target words and their acoustic semantic codes are tied to monotonized source boundaries and emitted together in chunks, so the model learns a read/wait/write policy without a separate speech-side controller. A two-stream architecture — a Thinker that plans target text and a lightweight Talker that predicts acoustic codes from the Thinker's hidden states — is shown to beat a matched unified decoder across latency tiers. If correct, the recipe lowers the data barrier for streaming S2ST from tens of thousands of hours to a few thousand, and matches or approaches a closed-source commercial system on RealSI while setting a strong frontier on ACL60/60-dev.

Core claim

The paper's claim, stated the way a sympathetic reader would state it: a speech language model can do sentence-level and long-form streaming S2ST with competitive quality–latency trade-offs from only about 2,000 hours of filtered paired S2ST data, provided three things are in place: auxiliary multitask pretraining anchors the model; the paired data are converted into a joint text-code commitment path in which each target word and its code span inherits a monotonized, source-aligned boundary and is emitted in one chunk; and text planning is architecturally separated from dense acoustic-code prediction. Offline S2ST becomes the special case of a single chunk; streaming is the general case. The

What carries the argument

The central object is the streaming trajectory τ = {(Y^text_c, Y^code_c, g_c)}: a chunk-wise schedule that says which source prefix must be read before the model may write both the target text chunk and its acoustic semantic codes. Boundaries come from forced alignment of source and target speech plus cross-lingual word alignment (SimAlign), monotonized to handle local reordering, then grouped into one-second read steps; latency tiers are induced by a sampled multiplier m ∈ {1,…,12} so one checkpoint covers all latencies. The second load-bearing mechanism is the Thinker–Talker factorization, p(Y^text_c | C_c) · p(Y^code_c | C_c, Y^text_c, H_θ), which routes code prediction through a lightwei

Load-bearing premise

The load-bearing premise is that the paired S2ST training signal — target speech synthesized by a TTS model, word boundaries from forced alignment, cross-lingual links from an alignment model, candidates kept by an ASR filter — is trustworthy enough that the commit times it encodes match real interpreting conditions; the paper's own Limitations section and Appendix J.2 (a forced-alignment error that unlocks a target word at 0.08 s before real content arrives) show where this

What would settle it

Re-run the identical recipe on naturally recorded simultaneous interpretation with ground-truth commit times. If streaming ASR-BLEU at the lowest latency tiers collapses — or if replacing the clean-reference alignment with an ASR-transcript alignment shifts a target word's unlock time from 0.08 s to ~0.96 s on disfluent speech, exactly the failure documented in Appendix J.2 — then the trajectory's low-latency benefit is not stable under alignment error. A complementary check: full-budget training across multiple seeds, since the reported seed stability covers only the cheaper 10%-without-aux a

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The paired-S2ST data requirement for streaming S2ST drops from tens of thousands of hours to about 2,000 — and the ablation shows keeping the auxiliary mixture matters more than the exact paired budget, with 10% of the paired data still competitive.
  • A single checkpoint can serve every latency tier, because training samples a latency multiplier over m∈{1,…,12}; deployment can trade latency against quality without retraining.
  • The unified-decoder baseline stays below the two-stream model at every tier even with identical trajectory supervision, which the paper reads as structural evidence that text planning and dense code prediction should be decoupled under data constraints.
  • Long-form, unbounded streaming does not collapse: on RealSI long-form S2ST the system stays competitive with LiveInterpret 2.0 from m4–m6, and on ACL60/60-dev it beats recent academic S2TT systems at comparable latency.
  • Offline S2ST is recovered as the C=1 special case, so the same recipe yields an offline model that matches UniSS(Q) on CVSS-T Text-BLEU/ASR-BLEU.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the trajectory is only as reliable as its alignments, the recipe's practical ceiling on real disfluent speech likely depends on the same failure mode the paper documents in its own Appendix J.2 — a forced-alignment error unlocking a target word at 0.08 s before real content arrives. A natural next step, which the paper leaves implicit, is to mine naturally recorded simultaneous interpretat
  • The joint text-code commitment idea is not tied to S2ST: any streaming generation task that must emit dense perceptual codes along a text-like plan — e.g., live dubbing with lip-sync constraints or streaming speech-to-music captioning — could reuse the same chunk-factorized objective.
  • The paper keeps the speech encoder and acoustic backend frozen; it states the chunk-wise flow-matching decoder is not fully streaming-native. If a streaming-native backend were adopted, the remaining boundary artifacts and the gap to the RTF<1 regime at low latency might close — a testable engineering extension.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes a training recipe for streaming speech-to-speech translation (S2ST) that uses only ~2k hours of paired cross-lingual S2ST data, layered atop large auxiliary multitask supervision. The main contributions are (1) a joint text-code trajectory supervision that schedules target text and acoustic semantic codes as a unified commitment path, and (2) a two-stream Thinker–Talker architecture that factorizes linguistic planning from dense acoustic code prediction. The system is evaluated on CVSS-T for offline S2ST, RealSI for sentence-level and long-form streaming S2ST/S2TT, and ACL60/60-dev for long-form S2TT, with ablations on data filtering, budget reduction, and latency-multiplier sampling. The authors report competitive quality–latency trade-offs, including matching a closed-source system (LiveInterpret 2.0) on ASR-BLEU in several operating regimes. The paper is an empirical training-recipe evaluation with multiple benchmarks, an ASR-backend sensitivity check, a seed-reproducibility proxy, and a small human/LLM-judge study.

Significance. If the results hold, this is a practically significant demonstration that careful trajectory construction, auxiliary multitask training, and a two-stream architecture can largely compensate for a very small paired-S2ST budget. The paper's strengths are its controlled matched-backbone comparison, the breadth of streaming evaluation (sentence-level and long-form, S2TT and S2ST, multiple latency metrics), the ASR-backend sensitivity analysis, the seed-variance proxy, and the unusually explicit disclosure of failure modes. The central risk is the unquantified reliability of the automatically constructed trajectories, since the paper's core contribution is explicitly a trajectory-based supervision method. The attached stress-test concern about forced-alignment/SimAlign errors is real and is not adequately resolved by the current filtering description.

major comments (2)
  1. [§3.2, §A.4, App. J.2] The core contribution is trajectory supervision, whose validity depends on force-aligned source/target word boundaries and SimAlign links. The paper's own Appendix J.2 shows a concrete failure: an FA error under transcript–audio mismatch unlocks a target word at 0.08 s, before real content arrives. The filtering described in §A.4 retains roughly 79–80% of candidates under ASR error-rate thresholds, but these thresholds check content, not commit-time correctness; no estimate is given of the residual fraction of trajectories with premature ready_times. If such trajectories are systematic, the learned streaming policy could appear artificially aggressive at low latency, which would directly affect the quality–latency comparison with LiveInterpret 2.0 in Table 3 and Figures 3–4. Please quantify the residual rate (e.g., by re-aligning a sample of retained trajectories with an ASR transcript a
  2. [§4.4, Table 3, Fig. 3, App. F] The headline "matching LiveInterpret 2.0" relies on a mixed measurement protocol: ASR-BLEU is computed locally from API-obtained audio under a unified pipeline, but the latency coordinate of the LiveInterpret operating point is the value reported in its paper (footnote 1), not measured under the same SimulEval/LAAL conditions used for the authors' own points. LAAL is sensitive to segmentation and simulator settings; the paper's Appendix F shows that when computation-aware latency is measured directly via the API, LiveInterpret shifts rightward by several seconds, changing the comparison. To support the abstract's claim, the main-text comparison should use latency values measured under a single protocol (or should present the Appendix F computation-aware comparison as the primary evidence).
minor comments (4)
  1. [§4.1, App. G.1] The main results in Table 3 and Figures 3–5 are from single training runs. The 3-seed reproducibility check is only for the cheaper 10% w/o aux proxy, which is reassuring but not a substitute for variance estimates at the main operating points. Please state this limitation explicitly in the main text and consider reporting at least one additional seed for the headline comparison.
  2. [Table 5 vs. Table 12] Table 5 reports Talker 10% w/o aux En→Zh m1 = 9.47, while Table 12 gives 9.04±0.54 for the same condition. These are close but not identical; clarify whether the numbers are from different runs or whether Table 5 uses a different decoding configuration, so readers can reconcile them.
  3. [Fig. 1] The trajectory-construction figure is dense. Consider explicitly labeling the time axis and the meaning of the color-coded brackets in the caption, since the monotonicization step is essential to the method.
  4. [App. H.1] The human listening study has N=15 per direction. The statistical tests and ICC are helpful, but this scale should be acknowledged as a limitation in the main text when human evidence is cited, not only in the appendix.

Circularity Check

0 steps flagged

No significant circularity; empirical recipe evaluated on held-out benchmarks; minor self-reliance on own components is not load-bearing.

full rationale

The paper's central claim is an empirical training recipe, not a derivation. The trajectory objective in Eq. (2) is constructed from forced alignments and SimAlign in Section 3.2, and the trained model is evaluated on held-out real recordings (RealSI, ACL60/60-dev, CVSS-T). No test-set target is used to fit any parameter, and no reported number is identical by construction to a fitted input: latency tiers are induced by the multiplier m, but ASR-BLEU, BLEU, A.PCP, and SIM-O are measured from generated outputs, and comparisons to LiveInterpret 2.0 and UniSS are external. The self-citations (DualCodec, FlexiVoice, Emilia) appear as fixed tokenizer, acoustic-backend, and data inputs rather than as evidence for the target S2ST claim, so they are ordinary self-reliance rather than circular justification. Appendix J.2 documents a real data-quality failure (a forced-alignment error unlocking a word at 0.08 s) and J.3 a premature-emission case; these are correctness and latency risks, not circular steps, and the paper's own mitigation via ASR re-transcription and filtering (Section A.4) is acknowledged rather than hidden. Overall, no load-bearing reduction to the paper's own inputs was found.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central claim rests on the quality of constructed trajectories (alignments, NIR filtering, ASR thresholds) and on the synthetic target speech being representative. These are training-pipeline choices rather than fitted physical constants, but the RANDOM vs NIR ablation shows that at least one of them (NIR filtering) materially changes the low-latency result. No new physical or architectural entities are postulated beyond the training constructs described in the method.

free parameters (5)
  • source chunk interval = 1 second
    Defines trajectory granularity in §3.2; all streaming tiers and the chunk-factorized objective depend on this discretization.
  • NIR difficulty and length quotas = difficulty {high:0.1, mid_high:0.3, mid_low:0.4, low:0.2}; length {short:0.1, medium:0.5, long:0.4}
    Used in A.3 to select training trajectories; the RANDOM vs NIR ablation in Table 4 shows a large m1 effect, so this choice is load-bearing for low-latency quality.
  • ASR pair filtering thresholds = 5% Chinese / 7% English
    A.4; sets which synthesized pairs enter training, retaining roughly 79–80% of candidates.
  • latency multiplier sampling parameters = a=0.18, b=0.017, Mmax=12; Gaussian mean=3.5, variance=4.0
    B.1; controls the distribution of read/write steps during streaming finetuning and defines the latency tiers used in evaluation.
  • stage data mixture ratio = ASR:S2TT:MT:TTS:S2ST = 0.2:1:0.5:1:1.5
    Stage 2 recipe; the data-efficiency claim depends on the presence of the auxiliary mixture, as shown by the 10% w/o aux ablation in Table 5.
axioms (4)
  • domain assumption Forced alignment and SimAlign give reliable word-level cross-lingual boundaries for committing target words.
    Section 3.2 Step 1; incorrect FA can unlock target words too early, as the paper's own J.2 bad-case shows.
  • domain assumption Target speech synthesized with OmniVoice under voice cloning is adequate supervision for learning semantic-code prediction on real target speech.
    A.2 and Limitations; the paired S2ST data is entirely synthesized rather than collected from natural simultaneous interpretation.
  • domain assumption Chunk-factorized teacher-forced log-likelihood (Eq. 2) is a valid surrogate for the streaming read/write policy.
    The model is trained on teacher trajectories and never trains on its own emitted audio; inference assumes that this transfers to committed, irrevocable streaming decisions.
  • domain assumption ASR-BLEU via Whisper-large-v3/Paraformer measures translation fidelity of synthesized speech.
    Appendix D.3; the paper's own G.2 probe with Qwen3-ASR-1.7B changes one of four settings, so the metric is not fully backend-invariant.

pith-pipeline@v1.3.0-alltime-deepseek · 49703 in / 11982 out tokens · 120120 ms · 2026-08-01T11:37:54.725676+00:00 · methodology

0 comments
read the original abstract

Long-form streaming speech-to-speech translation (S2ST) requires incremental, unbounded translation under strict latency constraints. Existing methods typically suffer from sentence-bounded supervision or demand massive paired-S2ST supervision. We introduce a training recipe enabling a speech language model for sentence-level and long-form streaming S2ST using only $\sim$2k hours of paired cross-lingual S2ST data, layered atop auxiliary supervision. Anchored by auxiliary multitask training, our approach remains robust even when the paired-S2ST budget itself is reduced by 90\%. Our core contribution, joint text-code trajectory supervision, schedules target text and acoustic semantic codes as a unified commitment path, eliminating the need for separate, unstable speech-side emission controllers. Furthermore, our two-stream Thinker--Talker factorization significantly outperforms unified-decoder baselines by decoupling linguistic reasoning from dense acoustic prediction to mitigate modality interference. Finally, our system achieves highly competitive quality-latency trade-offs on RealSI and ACL60/60-dev, matching state-of-the-art, closed-source S2ST systems such as LiveInterpret~2.0 on ASR-BLEU.

Figures

Figures reproduced from arXiv: 2607.19810 by Dekun Chen, Jiaqi Li, Mingjie Chen, Rongshen He, Xinyu Liang, Zhizheng Wu.

Figure 1
Figure 1. Figure 1: Streaming trajectory construction. Step 1: Word-level and cross-lingual alignments establish the earliest valid source prefix for each target word. Step 2: Target text and segmented target speech codes inherit monotonized boundaries and are grouped into discrete read/wait/write steps for joint emission. 3 Method 3.1 Problem Setup and Formulation A paired-audio S2ST dataset consists of N train￾ing samples, … view at source ↗
Figure 2
Figure 2. Figure 2: Matched backbone comparison. Thinker–Talker (left) and Dec-only (right) share an identical speech encoder, base LLM backbone (with independent LoRA adapters), semantic-code tokenizer Q, and frozen flow￾matching/vocoder backend. inherit the same monotonized source boundary t˜i . To form fixed-size streaming steps, we group adja￾cent target words and their codes whose boundaries fall within the same pre-defi… view at source ↗
Figure 3
Figure 3. Figure 3: RealSI sentence-level content trade-off. Dashed curves denote S2TT text BLEU, solid curves denote S2ST ASR-BLEU, and horizontal lines mark offline reference lines [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: RealSI sentence-level acoustic quality trade￾off: A.PCP and SIM-O against LAAL for En→Zh and Zh→En. trade-off. The two-stream Talker vastly outper￾forms both Seamless streaming and the matched Dec-only baseline across all latency tiers in both directions. Crucially, this massive ASR-BLEU lead confirms that trajectory finetuning alone can￾not resolve a unified decoder’s internal text/code conflict, cementin… view at source ↗
Figure 5
Figure 5. Figure 5: ACL60/60-dev long-form streaming S2TT En→Zh trade-off: BLEU against StreamLAAL for En→Zh and Zh→En. a tight, approaching margin on Zh→En. This con￾firms that the joint text-code commitment path ef￾fectively manages long-form target speech genera￾tion [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Pooled NIR and Spearman distributions for [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: RealSI long-form streaming S2TT trade-off for both directions. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: RealSI Sentence-level simultaneous S2ST under LAAL_CA. ASR-BLEU is plotted against computation [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: RealSI sentence-level simultaneous S2ST: active compute RTF vs. LAAL_CA. RTF counts only non-idle [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: RealSI long-form simultaneous S2TT under StreamLAAL_CA. [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: ACL60/60-dev Long-form simultaneous S2TT under StreamLAAL_CA. G.2 ASR Backend Sensitivity ASR-BLEU transcribes synthesized target speech with an ASR system and scores the transcript against the reference translation, so the metric can inherit systematic biases of the chosen ASR back￾end, particularly when the generated speech carries [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Anonymized rating interface used in the human listening study. Systems are randomly assigned to slots [PITH_FULL_IMAGE:figures/full_fig_p024_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: RealSI S2ST case study: En→Zh, m=1. Source um this ti this time uh uh uh this time he gives simeon the back of his hand um and not incidentally spreads his arms arms in a kind of ins in innocent spontaneous spontaneous um reminder reminder of the crucifixion in the future 0 2 4 6 8 10 12 14 16 Time (s) Target WAIT interval WAIT onset WAIT [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: RealSI S2ST case study: En→Zh, m=2. Source 0 5 10 15 20 Time (s) Target For example, this company has this kind of, uh, you know, uh, you've sold this product, but but, but it's like, even if your company isn'tbankrupt on its own, well, it's not going to affect WAIT interval WAIT onset WAIT [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: RealSI S2ST case study: Zh→En, m=1 [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: RealSI S2ST case study: Zh→En, m=2. Re-timed (concatenated) Original (continuous) [PITH_FULL_IMAGE:figures/full_fig_p027_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Boundary concatenation artifact (mel spectrogram). [PITH_FULL_IMAGE:figures/full_fig_p027_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: FA under transcript–audio mismatch (source clip; the false start [PITH_FULL_IMAGE:figures/full_fig_p029_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Premature emission / wrong write (Zh→En), source-consumed axis. Top: source. Middle (m=2): the agent commits “It’s called a podium.” at 6.0 s—before the disambiguating source (“sage”) arrives—then stalls in a WAIT. Bottom (m=4): the wider read window lets it hear the full phrase first and correctly emit “It’s called the sage on the stage.” [PITH_FULL_IMAGE:figures/full_fig_p029_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

300 extracted references · 17 canonical work pages · 10 internal anchors

  1. [1]

    Neural Information Processing Systems , year=

    Neural Ordinary Differential Equations , author=. Neural Information Processing Systems , year=

  2. [2]

    International Conference on Learning Representations , year=

    Flow Matching for Generative Modeling , author=. International Conference on Learning Representations , year=

  3. [3]

    2023 , eprint=

    Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision , author=. 2023 , eprint=

    2023

  4. [4]

    ArXiv , year=

    Language Models are Few-Shot Learners , author=. ArXiv , year=

  5. [5]

    Alex Nichol and Prafulla Dhariwal and Aditya Ramesh and Pranav Shyam and Pamela Mishkin and Bob McGrew and Ilya Sutskever and Mark Chen , booktitle=

  6. [6]

    ArXiv , year=

    Zero-Shot Text-to-Image Generation , author=. ArXiv , year=

  7. [7]

    ACM Computing Surveys , year=

    Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing , author=. ACM Computing Surveys , year=

  8. [8]

    ArXiv , year=

    Scaling Laws for Generative Mixed-Modal Language Models , author=. ArXiv , year=

  9. [9]

    ArXiv , year=

    Training Compute-Optimal Large Language Models , author=. ArXiv , year=

  10. [10]

    International Conference on Machine Learning , year=

    Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech , author=. International Conference on Machine Learning , year=

  11. [11]

    Kim, Jaehyeon and Kim, Sungwon and Kong, Jungil and Yoon, Sungroh , journal=

  12. [12]

    Rongjie Huang and Max W. Y. Lam and J. Wang and Dan Su and Dong Yu and Yi Ren and Zhou Zhao , booktitle=

  13. [13]

    Soong and Tao Qin and Sheng Zhao and Tie-Yan Liu , journal=

    Xu Tan and Jiawei Chen and Haohe Liu and Jian Cong and Chen Zhang and Yanqing Liu and Xi Wang and Yichong Leng and Yuanhao Yi and Lei He and Frank K. Soong and Tao Qin and Sheng Zhao and Tie-Yan Liu , journal=. 2022 , volume=

    2022

  14. [14]

    CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92) , author=

  15. [15]

    International Conference on Language Resources and Evaluation , year=

    Common Voice: A Massively-Multilingual Speech Corpus , author=. International Conference on Language Resources and Evaluation , year=

  16. [16]

    International Conference on Acoustics, Speech and Signal Processing , year=

    Librispeech: An ASR corpus based on public domain audio books , author=. International Conference on Acoustics, Speech and Signal Processing , year=

  17. [17]

    International Conference on Learning Representations , year=

    Hierarchical Generative Modeling for Controllable Speech Synthesis , author=. International Conference on Learning Representations , year=

  18. [18]

    Conference on Empirical Methods in Natural Language Processing , year=

    fairseq S ^2 : A Scalable and Integrable Speech Synthesis Toolkit , author=. Conference on Empirical Methods in Natural Language Processing , year=

  19. [19]

    International Conference on Machine Learning , year=

    Edresson Casanova and Julian Weber and Christopher Dane Shulby and Arnaldo C. International Conference on Machine Learning , year=

  20. [20]

    International Conference on Learning Representations , year=

    FastSpeech 2: Fast and High-Quality End-to-End Text to Speech , author=. International Conference on Learning Representations , year=

  21. [21]

    Vadim Popov and Ivan Vovk and Vladimir Gogoryan and Tasnima Sadekova and Mikhail Kudinov , booktitle=

  22. [22]

    Advances in Neural Information Processing Systems , year=

    Denoising diffusion probabilistic models , author=. Advances in Neural Information Processing Systems , year=

  23. [23]

    International Conference on Acoustics, Speech and Signal Processing , year=

    Jacob Kahn and Morgane Rivi. International Conference on Acoustics, Speech and Signal Processing , year=

  24. [24]

    ArXiv , year=

    MLS: A Large-Scale Multilingual Dataset for Speech Research , author=. ArXiv , year=

  25. [25]

    Kameoka and Takuhiro Kaneko and Kou Tanaka and Nobukatsu Hojo , journal=

    H. Kameoka and Takuhiro Kaneko and Kou Tanaka and Nobukatsu Hojo , journal=

  26. [26]

    ArXiv , year=

    The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods , author=. ArXiv , year=

    2018

  27. [27]

    Sequence-to-sequence modelling of

    Robinson, Carl and Obin, Nicolas and Roebel, Axel , booktitle=. Sequence-to-sequence modelling of

  28. [28]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , year=

    Textless speech emotion conversion using decomposed and discrete representations , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , year=

    2022

  29. [29]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing , volume=

    A regression approach to speech enhancement based on deep neural networks , author=. IEEE/ACM Transactions on Audio, Speech, and Language Processing , volume=. 2014 , publisher=

    2014

  30. [30]

    ArXiv , year=

    Real Time Speech Enhancement in the Waveform Domain , author=. ArXiv , year=

  31. [31]

    ArXiv , year=

    Universal Speech Enhancement with Score-based Diffusion , author=. ArXiv , year=

  32. [32]

    Advances in neural information processing systems , year=

    Transfer learning from speaker verification to multispeaker text-to-speech synthesis , author=. Advances in neural information processing systems , year=

  33. [33]

    International Conference on Machine Learning , year=

    Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis , author=. International Conference on Machine Learning , year=

  34. [34]

    ArXiv , year=

    Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder , author=. ArXiv , year=

  35. [35]

    Weiss and Mike Schuster and Navdeep Jaitly and Zongheng Yang and Z

    Jonathan Shen and Ruoming Pang and Ron J. Weiss and Mike Schuster and Navdeep Jaitly and Zongheng Yang and Z. Chen and Yu Zhang and Yuxuan Wang and R. J. Skerry-Ryan and Rif A. Saurous and Yannis Agiomyrgiannakis and Yonghui Wu , journal=. Natural

  36. [36]

    Transactions of the Association for Computational Linguistics , year=

    On Generative Spoken Language Modeling from Raw Audio , author=. Transactions of the Association for Computational Linguistics , year=

  37. [37]

    He Bai and Renjie Zheng and Junkun Chen and Xintong Li and Mingbo Ma and Liang Huang , booktitle=

  38. [38]

    Interspeech , year=

    Zal. Interspeech , year=

  39. [39]

    ArXiv , year=

    Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers , author=. ArXiv , year=

  40. [40]

    ArXiv , year=

    High Fidelity Neural Audio Compression , author=. ArXiv , year=

  41. [41]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing , year=

    SoundStream: An End-to-End Neural Audio Codec , author=. IEEE/ACM Transactions on Audio, Speech, and Language Processing , year=

  42. [42]

    IEEE/ACM Transactions on Audio, Speech, and Language Processing , year=

    HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units , author=. IEEE/ACM Transactions on Audio, Speech, and Language Processing , year=

  43. [43]

    ArXiv , year=

    Robust Speech Recognition via Large-Scale Weak Supervision , author=. ArXiv , year=

  44. [44]

    Annual Meeting of the Association for Computational Linguistics , year=

    Text-Free Prosody-Aware Generative Spoken Language Modeling , author=. Annual Meeting of the Association for Computational Linguistics , year=

  45. [45]

    Transactions of the Association for Computational Linguistics , year=

    Generative Spoken Dialogue Language Modeling , author=. Transactions of the Association for Computational Linguistics , year=

  46. [46]

    Interspeech , year=

    Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi , author=. Interspeech , year=

  47. [47]

    ArXiv , year=

    Attention is All you Need , author=. ArXiv , year=

  48. [48]

    ArXiv , year=

    Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation , author=. ArXiv , year=

  49. [49]

    CoRR , year=

    Adam: A Method for Stochastic Optimization , author=. CoRR , year=

  50. [50]

    Chen, Ricky T. Q. , title=. 2018 , url=

    2018

  51. [51]

    Journal of Computational and Applied Mathematics , year=

    A family of embedded Runge-Kutta formulae , author=. Journal of Computational and Applied Mathematics , year=

  52. [52]

    International Conference on Acoustics, Speech and Signal Processing , year=

    Fastpitch: Parallel text-to-speech with pitch prediction , author=. International Conference on Acoustics, Speech and Signal Processing , year=

  53. [53]

    ACM SIGGRAPH 2022 Conference Proceedings , year=

    Palette: Image-to-image diffusion models , author=. ACM SIGGRAPH 2022 Conference Proceedings , year=

    2022

  54. [54]

    International Conference on Acoustics, Speech and Signal Processing , year=

    Ribeiro, Fl. International Conference on Acoustics, Speech and Signal Processing , year=

  55. [55]

    Proceedings of IEEE pacific rim conference on communications computers and signal processing , volume=

    Mel-cepstral distance measure for objective speech quality assessment , author=. Proceedings of IEEE pacific rim conference on communications computers and signal processing , volume=. 1993 , organization=

    1993

  56. [56]

    international conference on machine learning , pages=

    Towards end-to-end prosody transfer for expressive speech synthesis with tacotron , author=. international conference on machine learning , pages=. 2018 , organization=

    2018

  57. [57]

    ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    SDR--half-baked or well done? , author=. ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2019 , organization=

    2019

  58. [58]

    arXiv preprint arXiv:1904.02882 , year=

    Libritts: A corpus derived from librispeech for text-to-speech , author=. arXiv preprint arXiv:1904.02882 , year=

    Pith/arXiv arXiv 1904

  59. [59]

    Language Resources and Evaluation , volume=

    TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese , author=. Language Resources and Evaluation , volume=. 2022 , publisher=

    2022

  60. [60]

    arXiv preprint arXiv:2005.08100 , year=

    Conformer: Convolution-augmented transformer for speech recognition , author=. arXiv preprint arXiv:2005.08100 , year=

    Pith/arXiv arXiv 2005

  61. [61]

    Yamamoto, Ryuichi and Song, Eunwoo and Kim, Jae-Min , booktitle=

  62. [62]

    Kevin Kilgour and Mauricio Zuluaga and Dominik Roblek and Matthew Sharifi , booktitle=. Fr

  63. [63]

    Interspeech , year=

    VoxCeleb2: Deep Speaker Recognition , author=. Interspeech , year=

  64. [64]

    Desplanques, Brecht and Thienpondt, Jenthe and Demuynck, Kris , booktitle=

  65. [65]

    arXiv preprint arXiv:2207.12598 , year=

    Classifier-free diffusion guidance , author=. arXiv preprint arXiv:2207.12598 , year=

    Pith/arXiv arXiv

  66. [66]

    Diffusion models beat

    Dhariwal, Prafulla and Nichol, Alexander , journal=. Diffusion models beat

  67. [67]

    arXiv preprint arXiv:1809.11096 , year=

    Large scale GAN training for high fidelity natural image synthesis , author=. arXiv preprint arXiv:1809.11096 , year=

    Pith/arXiv arXiv

  68. [68]

    Advances in neural information processing systems , volume=

    Glow: Generative flow with invertible 1x1 convolutions , author=. Advances in neural information processing systems , volume=

  69. [69]

    Advances in neural information processing systems , year=

    wav2vec 2.0: A framework for self-supervised learning of speech representations , author=. Advances in neural information processing systems , year=

  70. [70]

    Workshop on automatic speech recognition and understanding , year=

    The Kaldi speech recognition toolkit , author=. Workshop on automatic speech recognition and understanding , year=

  71. [71]

    Heusel, Martin and Ramsauer, Hubert and Unterthiner, Thomas and Nessler, Bernhard and Hochreiter, Sepp , journal=

  72. [72]

    Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and others , journal=

  73. [73]

    Park and William Chan and Yu Zhang and Chung-Cheng Chiu and Barret Zoph and Ekin Dogus Cubuk and Quoc V

    Daniel S. Park and William Chan and Yu Zhang and Chung-Cheng Chiu and Barret Zoph and Ekin Dogus Cubuk and Quoc V. Le , booktitle=

  74. [74]

    Acoustics, Speech, and Signal Processing, IEEE International Conference on , volume=

    SWITCHBOARD: Telephone speech corpus for research and development , author=. Acoustics, Speech, and Signal Processing, IEEE International Conference on , volume=. 1992 , organization=

    1992

  75. [75]

    arXiv preprint arXiv:2212.11377 , year=

    ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement , author=. arXiv preprint arXiv:2212.11377 , year=

    Pith/arXiv arXiv

  76. [76]

    Proceedings of the IEEE/CVF International conference on computer vision , pages=

    Artificial fingerprinting for generative models: Rooting deepfake attribution in training data , author=. Proceedings of the IEEE/CVF International conference on computer vision , pages=

  77. [77]

    Interspeech , year=

    Speech Resynthesis from Discrete Disentangled Self-Supervised Representations , author=. Interspeech , year=

  78. [78]

    IEEE Journal of Selected Topics in Signal Processing , volume=

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing , author=. IEEE Journal of Selected Topics in Signal Processing , volume=. 2022 , publisher=

    2022

  79. [79]

    Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022 , pages =

    Arun Babu and Changhan Wang and Andros Tjandra and Kushal Lakhotia and Qiantong Xu and Naman Goyal and Kritika Singh and Patrick von Platen and Yatharth Saraf and Juan Pino and Alexei Baevski and Alexis Conneau and Michael Auli , editor =. Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22...

    2022

  80. [80]

    arXiv preprint arXiv:2304.09116 , year=

    Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers , author=. arXiv preprint arXiv:2304.09116 , year=

    Pith/arXiv arXiv

Showing first 80 references.