REVIEW 5 cited by
GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Linear Recurrence has proven to be a powerful tool for modeling long sequences efficiently. In this work, we show that existing models fail to take full advantage of its potential. Motivated by this finding, we develop GateLoop, a foundational sequence model that generalizes linear recurrent models such as S4, S5, LRU and RetNet, by employing data-controlled state transitions. Utilizing this theoretical advance, GateLoop empirically outperforms existing models for auto-regressive language modeling. Our method comes with a low-cost $O(l)$ recurrent mode and an efficient $O(l \log_{2} l)$ parallel mode making use of highly optimized associative scan implementations. Furthermore, we derive an $O(l^2)$ surrogate attention mode, revealing remarkable implications for Transformer and recently proposed architectures. Specifically, we prove that our approach can be interpreted as providing data-controlled relative-positional information to Attention. While many existing models solely rely on data-controlled cumulative sums for context aggregation, our findings suggest that incorporating data-controlled complex cumulative products may be a crucial step towards more powerful sequence models.
Forward citations
Cited by 5 Pith papers
-
Let SSMs be ConvNets: State-space Modeling with Optimal Tensor Contractions
Treating state-space layers as tensor networks with CNN-style connectivity and optimized contraction orders yields hybrid SSM networks that outperform homogeneous SSMs on raw audio tasks and enable competitive streami...
-
Test-time regression: a unifying framework for designing sequence models with associative memory
Sequence models that perform associative recall are unified as 'test-time regression' layers, reducing to weighted, kernel, or online least-squares regression depending on three design choices.
-
Selective Attention: Enhancing Transformer through Principled Context Control
Selective Self-Attention adds query- and value-dependent temperature scaling to transformer attention, improving language modeling accuracy and passkey retrieval with under 0.5% extra parameters.
-
MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map
MetaLA removes the Key matrix from linear attention, adds input-dependent decay plus self-augmentation, and reports gains over many linear baselines.
-
Latent Mamba Operator for Partial Differential Equations
LaMO replaces attention in latent-token neural operators with bidirectional state-space models and reports consistent accuracy gains on six PDE benchmarks.
Discussion (0). Continue with ORCID to comment.