Convolutional Sequence to Sequence Learning
read the original abstract
The prevalent approach to sequence to sequence learning maps an input sequence to a variable length output sequence via recurrent neural networks. We introduce an architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training and optimization is easier since the number of non-linearities is fixed and independent of the input length. Our use of gated linear units eases gradient propagation and we equip each decoder layer with a separate attention module. We outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT'14 English-French translation at an order of magnitude faster speed, both on GPU and CPU.
This paper has not been read by Pith yet.
Forward citations
Cited by 13 Pith papers
-
Generating Long Sequences with Sparse Transformers
Sparse Transformers factorize attention to handle sequences tens of thousands long, achieving new SOTA density modeling on Enwik8, CIFAR-10, and ImageNet-64.
-
K-STEMIT: Knowledge-Informed Spatio-Temporal Efficient Multi-Branch Graph Neural Network for Subsurface Stratigraphy Thickness Estimation from Radar Data
K-STEMIT reduces RMSE by 21% for subsurface stratigraphy thickness estimation from radar data via a knowledge-informed spatio-temporal GNN with adaptive feature fusion and physical priors from the MAR weather model.
-
Progress Ratio Embeddings: An Impatience Signal for Robust Length Control in Neural Text Generation
Progress Ratio Embeddings use a trigonometric progress-ratio signal to deliver stable length control in transformers that generalizes to unseen target lengths.
-
YaRN: Efficient Context Window Extension of Large Language Models
YaRN extends the context window of RoPE-based LLMs like LLaMA more efficiently than prior methods, using 10x fewer tokens and 2.5x fewer steps while surpassing state-of-the-art performance and enabling extrapolation b...
-
Adaptive Federated Optimization
Proposes federated adaptive optimizers (FedAdagrad, FedAdam, FedYogi) with convergence analysis for non-convex objectives under data heterogeneity and reports empirical gains over FedAvg.
-
Time2Vec: Learning a Vector Representation of Time
Time2Vec learns a vector representation of time that improves model performance when used in place of raw time inputs across various models and problems.
-
Open-Ended Long-Form Video Question Answering via Hierarchical Convolutional Self-Attention Networks
Introduces HCSA, a hierarchical convolutional self-attention network for efficient long-form video QA with question-aware dependency modeling.
-
Joint Detection of Malicious Domains and Infected Clients
Sluice network transfer learning jointly detects infected clients and malicious domains from HTTPS traffic, outperforming separate models and identifying previously unknown threats.
-
Universal Transformers
Universal Transformers combine Transformer parallelism with recurrent updates and dynamic halting to achieve Turing-completeness under assumptions and outperform standard Transformers on algorithmic and language tasks.
-
Predicting the thermodynamics in the chromosphere from the translation of SDO data into the IRIS$^{2}$ inversion results using a visual transformer model
A visual transformer model trained on IRIS inversions predicts chromospheric temperature and density from SDO data with correlations around 0.8 on 80% of test cases.
-
Attention Is All You Need
Pith review generated a malformed one-line summary.
-
Deep Learning for Time Series Forecasting: The Electric Load Case
Compares feedforward, recurrent, sequence-to-sequence and temporal convolutional neural networks for short-term electric load forecasting through experiments on two real datasets.
-
Incremental Adaptation of NMT for Professional Post-editors: A User Study
User study with professional translators shows that incremental online adaptation of NMT reduces post-editing effort and improves translation quality.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.