Effective Approaches to Attention-based Neural Machine Translation
read the original abstract
An attentional mechanism has lately been used to improve neural machine translation (NMT) by selectively focusing on parts of the source sentence during translation. However, there has been little work exploring useful architectures for attention-based NMT. This paper examines two simple and effective classes of attentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. We demonstrate the effectiveness of both approaches over the WMT translation tasks between English and German in both directions. With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional systems which already incorporate known techniques such as dropout. Our ensemble model using different attention architectures has established a new state-of-the-art result in the WMT'15 English to German translation task with 25.9 BLEU points, an improvement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker.
This paper has not been read by Pith yet.
Forward citations
Cited by 20 Pith papers
-
Selective Contrastive Learning For Gloss Free Sign Language Translation
A pair selection strategy based on negative similarity dynamics strengthens contrastive supervision in gloss-free sign language translation by reducing noisy negatives.
-
The General Theory of Localization Methods
The localization method unifies kernel methods, local learning algorithms, MeanShift, Hopfield networks, and Transformers through local models, localization tricks, and hierarchical extensions.
-
Jet Quenching Identification via Supervised Learning in Simulated Heavy-Ion Collisions
Sequential machine learning on jet declustering history trees outperforms static models at identifying jet quenching in heavy-ion collision simulations.
-
Graph-based Knowledge Distillation by Multi-head Attention Network
Multi-head attention constructs a graph of dataset relations from the teacher embedding procedure and transfers it to the student via multi-task learning, yielding 7.05% higher CIFAR-100 accuracy than the student alon...
-
Creating A Neural Pedagogical Agent by Jointly Learning to Review and Assess
Bidirectional RNN with attention models real-time user knowledge from question-response sequences to predict correctness, outperforming baselines especially for new users on a large TOEIC mobile app dataset.
-
Leveraging Text Repetitions and Denoising Autoencoders in OCR Post-correction
Error distributions estimated from text repetitions enable training of denoising autoencoders that improve OCR post-correction on historical Finnish newspapers without manual training data.
-
Attention U-Net: Learning Where to Look for the Pancreas
Attention gates added to U-Net automatically focus on target organs in CT images and improve segmentation performance on abdominal datasets.
-
OGNet: Salient Object Detection with Output-guided Attention Module
OGNet proposes an output-guided attention module from multi-scale outputs and an intractable area F-measure loss to enhance salient object detection in edges and confusing areas while remaining lightweight.
-
Do Transformer Attention Heads Provide Transparency in Abstractive Summarization?
Analysis of transformer attention heads in abstractive summarization shows specialization in some heads and proposes a method to measure model reliance on learned attention distributions.
-
Attention Is All You Need
Pith review generated a malformed one-line summary.
-
Resource-Efficient CSI Prediction: A Gated Fusion and Factorized Projection Approach
A gated-fusion CSI predictor using GRU, attention, and DSLH reaches -13.84 dB NMSE with 26% fewer parameters and 2.3x higher throughput than a LinFormer baseline on 3GPP channels.
-
Skeleton-based Coherence Modeling in Narratives
Sentence-level models outperform skeleton-based approaches for narrative coherence despite a new SSN network improving on cosine and Euclidean baselines.
-
Automatically Learning Construction Injury Precursors from Text
Standard NLP classifiers can surface valid injury precursors from raw construction safety reports.
-
Investigating Self-Attention Network for Chinese Word Segmentation
Self-attention networks achieve competitive results to BiLSTM-CRF on Chinese word segmentation, with BERT and word integration yielding the best reported performance on six heterogeneous domain benchmarks.
-
Hierarchical Sequence to Sequence Voice Conversion with Limited Data
Hierarchical seq2seq model for parallel voice conversion pretrained as autoencoder on single-speaker data then adapted to limited multispeaker data, using mel spectrograms converted via wavenet vocoder.
-
Automatic Repair and Type Binding of Undeclared Variables using Neural Networks
Neural network trained on AST structural details repairs undeclared variable errors and infers types, reporting 81% success on location/identification and 80% on types for 1059 programs in the prutor dataset.
-
Improving Zero-shot Translation with Language-Independent Constraints
Language-independent constraints and regularization in multilingual Transformer NMT yield a 2.23 BLEU average gain on zero-shot pairs from the IWSLT 2017 dataset.
-
Positional Encoding in Transformer-Based Time Series Models: A Survey
A survey of positional encoding methods in transformer-based time series models that evaluates fixed, learnable, relative, and hybrid approaches on classification tasks and links effectiveness to data characteristics.
-
Gemma 2: Improving Open Language Models at a Practical Size
Gemma 2 models achieve leading performance at their sizes by combining established Transformer modifications with knowledge distillation for the 2B and 9B variants.
-
Predicting Drug Responses by Propagating Interactions through Text-Enhanced Drug-Gene Networks
A text-enhanced drug-gene network is constructed from articles and data, with edge embeddings estimated from cell line records to enable explainable drug sensitivity predictions at 94.74% accuracy.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.