Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Text

Jan \v{C}ernock\'y; Luk\'a\v{s} Burget; Murali Karthick Baskar; Ramon Astudillo; Shinji Watanabe; Takaaki Hori

RECORD

Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Text

Murali Karthick Baskar, Shinji Watanabe, Ramon Astudillo, Takaaki Hori, Lukáš Burget, Jan Černocký

Audio and Speech Processing · Submitted 30 Apr 2019

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim The method is able to leverage both unpaired speech and text data to outperform recently proposed related techniques in terms of \%WER.

how it is stated · asserted 0.99 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID1905.01152 (version 2)

Pith IDQLIUSP75, Pith's permanent record of this paper and its review

CitationJan \v{C}ernock\'y, Luk\'a\v{s} Burget, Murali Karthick Baskar, Ramon Astudillo, Shinji Watanabe, Takaaki Hori. "Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Text." arXiv:1905.01152 (2019). https://pith.science/paper/QLIUSP75

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,199 input tokens.

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.07 · human participants 0.03 · controlled comparison 0.57 · randomized assignment 0.01 · states quantitative result 0.15 · compares with prior results 0.96 · mentions code or data availability 0.98.

The abstract, sentence by sentence

arXiv listing abstract v2. Results and the main claim are underlined; show every role.

Sequence-to-sequence automatic speech recognition (ASR) models require large quantities of data to attain high performance. For this reason, there has been a recent surge in interest for unsupervised and semi-supervised training in such models. This work builds upon recent results showing notable improvements in semi-supervised training using cycle-consistency and related techniques. Such techniques derive training procedures and losses able to leverage unpaired speech and/or text data by combining ASR with Text-to-Speech (TTS) models. In particular, this work proposes a new semi-supervised loss combining an end-to-end differentiable ASR$\rightarrow$TTS loss with TTS$\rightarrow$ASR loss. The method is able to leverage both unpaired speech and text data to outperform recently proposed related techniques in terms of \%WER. We provide extensive results analyzing the impact of data quantity and speech and text modalities and show consistent gains across WSJ and Librispeech corpora. Our code is provided in ESPnet to reproduce the experiments.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Sequence-to-sequence automatic speech recognition (ASR) models require large quantities of data to attain high performance.	background 1.00	0.06	0.02	asserted 1.00
s1	For this reason, there has been a recent surge in interest for unsupervised and semi-supervised training in such models.	background 1.00	0.03	0.02	asserted 1.00
s2	This work builds upon recent results showing notable improvements in semi-supervised training using cycle-consistency and related techniques.	background 0.93	0.08	0.04	asserted 0.99
s3	Such techniques derive training procedures and losses able to leverage unpaired speech and/or text data by combining ASR with Text-to-Speech (TTS) models.	background 0.95	0.09	0.04	asserted 1.00
s4	In particular, this work proposes a new semi-supervised loss combining an end-to-end differentiable ASR$\rightarrow$TTS loss with TTS$\rightarrow$ASR loss.	contribution 0.69	0.93	0.17	asserted 1.00
s5	The method is able to leverage both unpaired speech and text data to outperform recently proposed related techniques in terms of \%WER.	result 0.98	0.94	0.78	asserted 0.99
s6	We provide extensive results analyzing the impact of data quantity and speech and text modalities and show consistent gains across WSJ and Librispeech corpora.	result 0.84	0.97	0.19	asserted 1.00
s7	Our code is provided in ESPnet to reproduce the experiments.	contribution 0.91	0.76	0.02	asserted 0.99

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99.

What was examined, and what was not

sources read

arXiv listing abstract v2

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Sequence-to-sequence automatic speech recognition (ASR) models require large quantities of data to attain high performance. For this reason, there has been a recent surge in interest for unsupervised and semi-supervised training in such models. This work builds upon recent results showing notable improvements in semi-supervised training using cycle-consistency and related techniques. Such techniques derive training procedures and losses able to leverage unpaired speech and/or text data by combining ASR with Text-to-Speech (TTS) models. In particular, this work proposes a new semi-supervised loss combining an end-to-end differentiable ASR$\rightarrow$TTS loss with TTS$\rightarrow$ASR loss. The method is able to leverage both unpaired speech and text data to outperform recently proposed related techniques in terms of \%WER. We provide extensive results analyzing the impact of data quantity and speech and text modalities and show consistent gains across WSJ and Librispeech corpora. Our code is provided in ESPnet to reproduce the experiments.

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

16 arXiv papers cite this in 19 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 11 sentences without a confident label

Compares with it 3 papers

The recent work [ref] which uses similar data partition for semi-supervised learning with attention model is also included.

Data Techniques For Online End-to-end Speech Recognition · Summary on WSJ · 0.96
However, there are important differences compared with previous ASR-TTS self-supervised learning methods ( [ref] , [ref] ).

10 hours data is all you need · Related work · 0.85
The recent work [ref] which uses the same data partition for semi-supervised learning with attention models is also included.

Semi-supervised ASR by End-to-end Self-training · Results summary · 0.97

Cites it as background 5 papers

Many algorithms exist which incorporate unlabelled data to improve ASR in the low-resource setting, including representation learning [ref] , pseudo-labeling [ref] , local prior matching [ref] , pseudo-label augmentation [ref] , adversarial training [ref] and back translation [ref] .

Iterative Pseudo-Labeling for Speech Recognition · Introduction · 0.85
One group of work relies on the TTS models and self/semi-supervised learning techniques to exploit unpaired speech and text data in the ASR training process [ref] .

Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition · Related Work · 0.92
Typical examples include the classic language model rescoring methods [ref] , training an additional text-to-speech model to produce pseudo paired speech-text data [ref] , and achieving adversarial learning by training an extra criticizing language model [ref] .

Sequence-to-sequence Automatic Speech Recognition with Word Embedding Regularization and Fused Decoding · Introduction · 0.93

What it cites · 3 sentences without a confident label

Builds on 1 paper

This improves the TTE cycle loss in [ref] and is related to the straight-through loss in [ref] * We complement this with a text-only pipeline loss using TTS$→$ASR.

End-to-End Feedback Loss in Speech Chain Framework via Straight-Through Estimator · Introduction · 0.95

Uses the method or tool of 1 paper

The CE objective is optimized using AdaDelta [ref] with an initial learning rate set to $1.0$.

ADADELTA: An Adaptive Learning Rate Method · Experimental Setup · 1.00

Compares with 1 paper

Note that the performance on WSJ-SI284 is inferior to our previously reported baseline [ref] .

Promising Accurate Prefix Boosting for sequence-to-sequence ASR · Model comparison with literature · 0.89

Cites as background 2 papers

In this category, [ref] proposes an end-to-end differentiable loss integrating ASR and TTS models by the straight-through estimator.

End-to-End Feedback Loss in Speech Chain Framework via Straight-Through Estimator · Introduction · 0.97
Sequence-to-sequence (seq2seq) ASR training directly maps a speech input to an output character sequence using a neural network [ref] , similar to those used in machine translation [ref] .

Neural Machine Translation by Jointly Learning to Align and Translate · Introduction · 1.00

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.