A GPT's backward pass through attention computes a "Reversed Attention" matrix, which can rank attention heads and, when injected into the forward pass, improve task accuracy without weight updates.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Reversed Attention: On The Gradient Descent Of Attention Layers In GPT
A GPT's backward pass through attention computes a "Reversed Attention" matrix, which can rank attention heads and, when injected into the forward pass, improve task accuracy without weight updates.