RECORD
Efficient Encoder-Decoder Transformer Decoding for Decomposable Tasks
Computation and Language · Submitted 19 Mar 2024
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We achieve computation reduction that roughly scales with the number of subtasks, gaining up to 4.6x speed-up over state-of-the-art models for dialogue state tracking, summarization, and question-answering tasks, with comparable or better performance.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Transformer-based NLP models are powerful but have high computational costs that limit deployment. Finetuned encoder-decoder models are popular in specialized domains and can outperform larger more generalized decoder-only models, such as GPT-4. We introduce a new configuration for encoder-decoder models that improves efficiency on structured output and decomposable tasks where multiple outputs are required for a single shared input. Our method, prompt-in-decoder (PiD), encodes the input once and decodes the output in parallel, boosting both training and inference efficiency by avoiding duplicate input encoding and increasing the operational intensity (ratio of numbers of arithmetic operation to memory access) of decoding process by sharing the input key-value cache. We achieve computation reduction that roughly scales with the number of subtasks, gaining up to 4.6x speed-up over state-of-the-art models for dialogue state tracking, summarization, and question-answering tasks, with comparable or better performance.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Transformer-based NLP models are powerful but have high computational costs that limit deployment. | background 1.00 | 0.05 | 0.02 | asserted 1.00 |
| s1 | Finetuned encoder-decoder models are popular in specialized domains and can outperform larger more generalized decoder-only models, such as GPT-4. | background 1.00 | 0.06 | 0.03 | asserted 0.84 |
| s2 | We introduce a new configuration for encoder-decoder models that improves efficiency on structured output and decomposable tasks where multiple outputs are required for a single shared input. | contribution 1.00 | 0.75 | 0.30 | asserted 1.00 |
| s3 | Our method, prompt-in-decoder (PiD), encodes the input once and decodes the output in parallel, boosting both training and inference efficiency by avoiding duplicate input encoding and increasing the operational intensity (ratio of numbers of arithmetic operation to memory access) of decoding process by sharing the input key-value cache. | method 0.95 | 0.83 | 0.21 | asserted 1.00 |
| s4 | We achieve computation reduction that roughly scales with the number of subtasks, gaining up to 4.6x speed-up over state-of-the-art models for dialogue state tracking, summarization, and question-answering tasks, with comparable or better performance. | result 1.00 | 0.96 | 0.71 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v3
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Transformer-based NLP models are powerful but have high computational costs that limit deployment. Finetuned encoder-decoder models are popular in specialized domains and can outperform larger more generalized decoder-only models, such as GPT-4. We introduce a new configuration for encoder-decoder models that improves efficiency on structured output and decomposable tasks where multiple outputs are required for a single shared input. Our method, prompt-in-decoder (PiD), encodes the input once and decodes the output in parallel, boosting both training and inference efficiency by avoiding duplicate input encoding and increasing the operational intensity (ratio of numbers of arithmetic operation to memory access) of decoding process by sharing the input key-value cache. We achieve computation reduction that roughly scales with the number of subtasks, gaining up to 4.6x speed-up over state-of-the-art models for dialogue state tracking, summarization, and question-answering tasks, with comparable or better performance.
Human review
How the literature uses it
1 arXiv paper cite this in 3 sentences
Papers citing it · 1 sentence without a confident label
Compares with it 1 paper
-
Finally, [ref] propose a method to encode input text once and have it be shared across each decoder prompt and apply their method to RadQA to obtain similar EM as the best T5 model shown in [ref] .
TOP-Training: Target-Oriented Pretraining for Medical Extractive Question Answering · Related Work · 0.84
What it cites · 5 sentences without a confident label
Cites as support 1 paper
-
For summarization and information extraction, decomposing a long target output into multiple shorter sequences mitigates attention degeneration issues [ref] , leading to a boost in accuracy and efficiency.
Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder · Introduction · 0.99
Cites as background 7 papers
-
Multi-query attention [ref] and grouped-query attention [ref] modify transformer architectures to employ a single (or fewer) key-value attention head for multiple query heads to reduce the memory access.
Fast Transformer Decoding: One Write-Head is All You Need · Related Work · 0.98
-
To reduce costs, researchers have investigated multiple approaches, including: i) model compression (e.g., distillation [ref] , quantization [ref] , and mixture of experts [ref] ); ii) decoding strategy (e.g., speculative decoding [ref] and parallel decoding [ref] ); iii) algorithm-level attention optimizations (e.g., sparse attention [ref] , fewer number of key-value heads [ref] and Hydragen [ref] ); iv) kernel-level attention optimizations (e.g., FlashAttention [ref] and FlashInfer [ref] ); and v) memory management (e.g. paged attention [ref] and radix attention [ref] ).
Knowledge Distillation: A Survey · Introduction · 0.97
-
To reduce costs, researchers have investigated multiple approaches, including: i) model compression (e.g., distillation [ref] , quantization [ref] , and mixture of experts [ref] ); ii) decoding strategy (e.g., speculative decoding [ref] and parallel decoding [ref] ); iii) algorithm-level attention optimizations (e.g., sparse attention [ref] , fewer number of key-value heads [ref] and Hydragen [ref] ); iv) kernel-level attention optimizations (e.g., FlashAttention [ref] and FlashInfer [ref] ); and v) memory management (e.g. paged attention [ref] and radix attention [ref] ).
Accelerating Large Language Model Decoding with Speculative Sampling · Introduction · 0.97
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.