REVIEW 8 cited by
Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation
read the original abstract
We study the power of cross-attention in the Transformer architecture within the context of transfer learning for machine translation, and extend the findings of studies into cross-attention when training from scratch. We conduct a series of experiments through fine-tuning a translation model on data where either the source or target language has changed. These experiments reveal that fine-tuning only the cross-attention parameters is nearly as effective as fine-tuning all parameters (i.e., the entire translation model). We provide insights into why this is the case and observe that limiting fine-tuning in this manner yields cross-lingually aligned embeddings. The implications of this finding for researchers and practitioners include a mitigation of catastrophic forgetting, the potential for zero-shot translation, and the ability to extend machine translation models to several new language pairs with reduced parameter storage overhead.
Forward citations
Cited by 8 Pith papers
-
Value-Decomposed Reinforcement Learning Framework for Taxiway Routing with Hierarchical Conflict-Aware Observations
CaTR applies value-decomposed RL with hierarchical conflict-aware observations to achieve better safety-efficiency trade-offs than planning, optimization, and standard RL baselines in a realistic airport taxiway simulation.
-
Generalizable Radio-Frequency Radiance Fields for Spatial Spectrum Synthesis
GRaF learns a scene-independent latent RF radiance field from proximate transmitters via an interpolation theory, then uses neural ray tracing to synthesize spectra at new transmitter or receiver positions.
-
One Prompt, Many Sounds: Modeling Listener Variability in LLM-Based Equalization
LLMs using in-context learning and fine-tuning on listener experiment data generate equalization settings that align better with population preferences than random sampling or static presets.
-
One Prompt, Many Sounds: Modeling Listener Variability in LLM-Based Equalization
LLM-based equalizers can match the spread of 11 human listeners' EQ choices for text prompts better than random sampling, but are not yet perceptually validated.
-
Causal Fingerprints of AI Generative Models
Proposes causal fingerprints via causality-decoupling in pre-trained diffusion residual latent space for improved source attribution across GANs and diffusion models.
-
Time-Conditioned and Multi-Time Survival Prediction from 2D PET/CT Projections in Lung Cancer
ATCS and MTS models report mean time-dependent AUCs of 0.794 and 0.793 versus 0.767 for baseline TCS on held-out test data from 292 NSCLC patients.
-
Value-Decomposed Reinforcement Learning Framework for Taxiway Routing with Hierarchical Conflict-Aware Observations
CaTR is a value-decomposed RL framework with hierarchical conflict-aware observations that reports better safety-efficiency trade-offs than planning, optimization, and RL baselines on a simulated Changsha Huanghua air...
-
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.