A Neural Transducer

David Sussillo; Ilya Sutskever; Navdeep Jaitly; Oriol Vinyals; Quoc V. Le; Samy Bengio

RECORD

A Neural Transducer

Navdeep Jaitly, David Sussillo, Quoc V. Le, Oriol Vinyals, Ilya Sutskever, Samy Bengio

Machine Learning · Submitted 16 Nov 2015

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Our experiments show that the Neural Transducer works well in settings where it is required to produce output predictions as data come in.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,978 input tokens.

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.57 · human participants 0.04 · controlled comparison 0.20 · randomized assignment 0.01 · states quantitative result 0.04 · compares with prior results 0.55 · mentions code or data availability 0.02.

The abstract, sentence by sentence

arXiv listing abstract v4. Results and the main claim are underlined; show every role.

Sequence-to-sequence models have achieved impressive results on various tasks. However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences. This is because they generate an output sequence conditioned on an entire input sequence. In this paper, we present a Neural Transducer that can make incremental predictions as more input arrives, without redoing the entire computation. Unlike sequence-to-sequence models, the Neural Transducer computes the next-step distribution conditioned on the partially observed input sequence and the partially generated sequence. At each time step, the transducer can decide to emit zero to many output symbols. The data can be processed using an encoder and presented as input to the transducer. The discrete decision to emit a symbol at every time step makes it difficult to learn with conventional backpropagation. It is however possible to train the transducer by using a dynamic programming algorithm to generate target discrete decisions. Our experiments show that the Neural Transducer works well in settings where it is required to produce output predictions as data come in. We also find that the Neural Transducer performs well for long sequences even when attention mechanisms are not used.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Sequence-to-sequence models have achieved impressive results on various tasks.	background 1.00	0.06	0.02	asserted 1.00
s1	However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences.	background 0.73	0.12	0.04	asserted 1.00
s2	This is because they generate an output sequence conditioned on an entire input sequence.	background 0.95	0.07	0.03	asserted 1.00
s3	In this paper, we present a Neural Transducer that can make incremental predictions as more input arrives, without redoing the entire computation.	contribution 1.00	0.87	0.40	asserted 1.00
s4	Unlike sequence-to-sequence models, the Neural Transducer computes the next-step distribution conditioned on the partially observed input sequence and the partially generated sequence.	method 0.95	0.55	0.13	asserted 1.00
s5	At each time step, the transducer can decide to emit zero to many output symbols.	method 0.95	0.35	0.05	asserted 0.82
s6	The data can be processed using an encoder and presented as input to the transducer.	method 1.00	0.23	0.03	hedged possibility 0.64
s7	The discrete decision to emit a symbol at every time step makes it difficult to learn with conventional backpropagation.	limitation 0.44	0.25	0.02	asserted 0.98
s8	It is however possible to train the transducer by using a dynamic programming algorithm to generate target discrete decisions.	method 0.98	0.44	0.05	hedged possibility 0.88
s9	Our experiments show that the Neural Transducer works well in settings where it is required to produce output predictions as data come in.	result 1.00	0.96	0.47	asserted 1.00
s10	We also find that the Neural Transducer performs well for long sequences even when attention mechanisms are not used.	result 1.00	0.96	0.14	asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.98.

What was examined, and what was not

sources read

arXiv listing abstract v4

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Sequence-to-sequence models have achieved impressive results on various tasks. However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences. This is because they generate an output sequence conditioned on an entire input sequence. In this paper, we present a Neural Transducer that can make incremental predictions as more input arrives, without redoing the entire computation. Unlike sequence-to-sequence models, the Neural Transducer computes the next-step distribution conditioned on the partially observed input sequence and the partially generated sequence. At each time step, the transducer can decide to emit zero to many output symbols. The data can be processed using an encoder and presented as input to the transducer. The discrete decision to emit a symbol at every time step makes it difficult to learn with conventional backpropagation. It is however possible to train the transducer by using a dynamic programming algorithm to generate target discrete decisions. Our experiments show that the Neural Transducer works well in settings where it is required to produce output predictions as data come in. We also find that the Neural Transducer performs well for long sequences even when attention mechanisms are not used.

Human review

How Pith checks papers → · Full audit of this paper →

References

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Language Models as Knowledge Bases?
cs.CL 2019-09 accept novelty 7.0 of 10

BERT stores relational knowledge extractable via cloze queries without fine-tuning and matches supervised baselines on open-domain QA tasks.
Recent Advances in Recurrent Neural Networks
cs.NE 2017-12 conditional novelty 2.0 of 10

A narrative survey that organizes the standard RNN, LSTM, GRU, and memory-network literature into training methods, architectures, regularization techniques, and signal-processing applications.

How the literature uses it

20 arXiv papers cite this in 27 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 14 sentences without a confident label

Cites it as background 10 papers

To make AED models streamable, various methods have been proposed: neural transducer (NT) [ref] , triggered attention [ref] , adaptive computation steps (ACS) [ref] , continuous integrate-and-fire (CIF) [ref] , streaming chunk-aware multi-head attention (SCAMA) [ref] , local windowing [ref] , Gaussian mixture model (GMM) attention [ref] , neural autoregressive transducer (NAT) [ref] , and monotonic chunkwise attention (MoChA) [ref] .

Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition · Introduction · 0.95
To address this, several novel streaming attention-based S2S models have been proposed including local windowing methods [ref] , neural transducer [ref] , hard monotonic attention [ref] , adaptive computation steps [ref] , triggered attention [ref] , and continuous integrate-and-fire [ref] .

Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR · Introduction · 0.90
Finally, Jaitly et al. [ref] used an online sequence-to-sequence method with conditioning on partial inputs, which yielded encouraging results on the TIMIT dataset.

An online sequence-to-sequence model for noisy speech recognition · Relation To Prior Work · 0.95

What it cites · 2 sentences without a confident label

Contests 1 paper

[ref] attempt to rectify this for speech recognition by using a moving windowed attention, but they do not provide a mechanism to address the situation that arises when no output can be produced from the windowed segment of data.

End-to-End Attention-based Large Vocabulary Speech Recognition · Related Work · contest 0.79

Uses the method or tool of 1 paper

We first describe how the context vector is computed by an attention model similar to earlier work [ref] .

Listen, Attend and Spell · Computing $f_context$ · 0.93

Cites as background 6 papers

The recently introduced sequence-to-sequence model has shown success in many tasks that map sequences to sequences, e.g., translation, speech recognition, image captioning and dialogue modeling [ref] .

A Neural Network Approach to Context-Sensitive Generation of Conversational Responses · Introduction · 0.99
The recently introduced sequence-to-sequence model has shown success in many tasks that map sequences to sequences, e.g., translation, speech recognition, image captioning and dialogue modeling [ref] .

Listen, Attend and Spell · Introduction · 0.99
In the past few years, many proposals have been made to add more power or flexibility to neural networks, especially via the concept of augmented memory [ref] or augmented arithmetic units [ref] .

Neural Programmer: Inducing Latent Programs with Gradient Descent · Related Work · 1.00

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.